root / docs / admin-guide.rst @ 60c6bb5e
History | View | Annotate | Download (105.5 kB)
1 |
.. _admin-guide: |
---|---|
2 |
|
3 |
Synnefo Administrator's Guide |
4 |
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
5 |
|
6 |
This is the complete Synnefo Administrator's Guide. |
7 |
|
8 |
|
9 |
.. _syn+archip: |
10 |
|
11 |
General Synnefo Architecture |
12 |
============================ |
13 |
|
14 |
The following figure shows a detailed view of the whole Synnefo architecture |
15 |
and how it interacts with multiple Ganeti clusters. We hope that after reading |
16 |
the Administrator's Guide you will be able to understand every component and |
17 |
all the interactions between them. |
18 |
|
19 |
.. image:: images/synnefo-arch2.png |
20 |
:width: 100% |
21 |
:target: _images/synnefo-arch2.png |
22 |
|
23 |
Synnefo also supports RADOS as an alternative storage backend for |
24 |
Files/Images/VM disks. You will find the :ref:`corresponding figure |
25 |
<syn+archip+rados>` later in this guide. |
26 |
|
27 |
|
28 |
Identity Service (Astakos) |
29 |
========================== |
30 |
|
31 |
|
32 |
Authentication methods |
33 |
---------------------- |
34 |
|
35 |
Astakos supports multiple authentication methods: |
36 |
|
37 |
* local username/password |
38 |
* LDAP / Active Directory |
39 |
* SAML 2.0 (Shibboleth) federated logins |
40 |
|
41 |
|
42 |
|
43 |
|
44 |
.. _shibboleth-auth: |
45 |
|
46 |
Shibboleth Authentication |
47 |
~~~~~~~~~~~~~~~~~~~~~~~~~ |
48 |
|
49 |
Astakos can delegate user authentication to a Shibboleth federation. |
50 |
|
51 |
To setup shibboleth, install package:: |
52 |
|
53 |
apt-get install libapache2-mod-shib2 |
54 |
|
55 |
Change appropriately the configuration files in ``/etc/shibboleth``. |
56 |
|
57 |
Add in ``/etc/apache2/sites-available/synnefo-ssl``:: |
58 |
|
59 |
ShibConfig /etc/shibboleth/shibboleth2.xml |
60 |
Alias /shibboleth-sp /usr/share/shibboleth |
61 |
|
62 |
<Location /ui/login/shibboleth> |
63 |
AuthType shibboleth |
64 |
ShibRequireSession On |
65 |
ShibUseHeaders On |
66 |
require valid-user |
67 |
</Location> |
68 |
|
69 |
and before the line containing:: |
70 |
|
71 |
ProxyPass / http://localhost:8080/ retry=0 |
72 |
|
73 |
add:: |
74 |
|
75 |
ProxyPass /Shibboleth.sso ! |
76 |
|
77 |
Then, enable the shibboleth module:: |
78 |
|
79 |
a2enmod shib2 |
80 |
|
81 |
After passing through the apache module, the following tokens should be |
82 |
available at the destination:: |
83 |
|
84 |
eppn # eduPersonPrincipalName |
85 |
Shib-InetOrgPerson-givenName |
86 |
Shib-Person-surname |
87 |
Shib-Person-commonName |
88 |
Shib-InetOrgPerson-displayName |
89 |
Shib-EP-Affiliation |
90 |
Shib-Session-ID |
91 |
|
92 |
Astakos keeps a map of shibboleth users using the value of the ``REMOTE_USER`` |
93 |
header, passed by the ``mod_shib2`` module. This happens in order to be able to |
94 |
identify the astakos account the shibboleth user is associated to, every time |
95 |
the user logs in from an affiliate shibboleth IdP. |
96 |
|
97 |
The shibboleth attribute which gets mapped to the ``REMOTE_USER`` header can be |
98 |
changed in ``/etc/shibboleth/shibboleth2.xml`` configuration file. |
99 |
|
100 |
.. code-block:: xml |
101 |
|
102 |
<!-- The ApplicationDefaults element is where most of Shibboleth's SAML bits are defined. --> |
103 |
<ApplicationDefaults entityID="https://sp.example.org/shibboleth" |
104 |
REMOTE_USER="eppn persistent-id targeted-id"> |
105 |
|
106 |
.. warning:: |
107 |
|
108 |
Changing ``mod_shib2`` ``REMOTE_USER`` to map to different shibboleth |
109 |
attributes will probably invalidate any existing shibboleth enabled users in |
110 |
astakos database. Those users won't be able to login to their existing accounts. |
111 |
|
112 |
|
113 |
Finally, add 'shibboleth' in ``ASTAKOS_IM_MODULES`` list. The variable resides |
114 |
inside the file ``/etc/synnefo/20-snf-astakos-app-settings.conf`` |
115 |
|
116 |
Twitter Authentication |
117 |
~~~~~~~~~~~~~~~~~~~~~~ |
118 |
|
119 |
To enable twitter authentication while signed in under a Twitter account, |
120 |
visit dev.twitter.com/apps. |
121 |
|
122 |
Click Create an application. |
123 |
|
124 |
Fill the necessary information and for callback URL give:: |
125 |
|
126 |
https://node1.example.com/ui/login/twitter/authenticated |
127 |
|
128 |
Finally, add 'twitter' in ``ASTAKOS_IM_MODULES`` list. The variable resides |
129 |
inside the file ``/etc/synnefo/20-snf-astakos-app-settings.conf`` |
130 |
|
131 |
Google Authentication |
132 |
~~~~~~~~~~~~~~~~~~~~~ |
133 |
|
134 |
To enable google authentication while signed in under a Google account, |
135 |
visit https://code.google.com/apis/console/. |
136 |
|
137 |
Under API Access select Create another client ID, select Web application, |
138 |
expand more options in Your site or hostname section and in Authorized |
139 |
Redirect URIs add: |
140 |
|
141 |
|
142 |
Fill the necessary information and for callback URL give:: |
143 |
|
144 |
https://node1.example.com/ui/login/google/authenticated |
145 |
|
146 |
Finally, add 'google' in ``ASTAKOS_IM_MODULES`` list. The variable resides |
147 |
inside the file ``/etc/synnefo/20-snf-astakos-app-settings.conf`` |
148 |
|
149 |
|
150 |
Working with Astakos |
151 |
-------------------- |
152 |
|
153 |
User registration |
154 |
~~~~~~~~~~~~~~~~~ |
155 |
|
156 |
When a new user signs up, he/she is not directly marked as active. You can see |
157 |
his/her state by running (on the machine that runs the Astakos app): |
158 |
|
159 |
.. code-block:: console |
160 |
|
161 |
$ snf-manage user-list |
162 |
|
163 |
More detailed user status is provided in the `status` field of the `user-show` |
164 |
command: |
165 |
|
166 |
.. code-block:: console |
167 |
|
168 |
$ snf-manage user-show <user-id> |
169 |
|
170 |
id : 6 |
171 |
uuid : 78661411-5eed-412f-a9ea-2de24f542c2e |
172 |
status : Accepted/Active (accepted policy: manual) |
173 |
email : user@synnefo.org |
174 |
.... |
175 |
|
176 |
Based on the `astakos-app` configuration, there are several ways for a user to |
177 |
get verified and activated in order to be able to login. We discuss the user |
178 |
verification and activation flow in the following section. |
179 |
|
180 |
User activation flow |
181 |
```````````````````` |
182 |
|
183 |
A user can register for an account using the astakos signup form. Once the form |
184 |
is submited successfully a user entry is created in astakos database. That entry |
185 |
is passed through the astakos activation backend which handles whether the user |
186 |
should be automatically verified and activated. |
187 |
|
188 |
Email verification |
189 |
`````````````````` |
190 |
|
191 |
The verification process takes place in order to ensure that the user owns the |
192 |
email provided during the signup process. By default, after each successful |
193 |
signup astakos notifies user with an verification url via email. |
194 |
|
195 |
At this stage: |
196 |
|
197 |
* subsequent registrations invalidate and delete the previous registrations |
198 |
of the same email address. |
199 |
|
200 |
* in case user misses the initial notification, additional emails can be |
201 |
send either via the url which is prompted to the user if he tries to |
202 |
login, or by the administrator using the ``snf-manage user-activation-send |
203 |
<userid>`` command. |
204 |
|
205 |
* administrator may also enforce a user to get verified using the |
206 |
``snf-manage user-modify --verify <userid>`` command. |
207 |
|
208 |
Account activation |
209 |
`````````````````` |
210 |
|
211 |
Once the user gets verified, it is time for Astakos to decide whether or not to |
212 |
proceed through user activation process. If ``ASTAKOS_MODERATION_ENABLED`` |
213 |
setting is set to ``False`` (default value) user gets activated automatically. |
214 |
|
215 |
In case the moderation is enabled Astakos may still automatically activate the |
216 |
user in the following cases: |
217 |
|
218 |
* User email matches any of the regular expressions defined in |
219 |
``ASTAKOS_RE_USER_EMAIL_PATTERNS`` (defaults to ``[]``) |
220 |
* User used a signup method (e.g. ``shibboleth``) for which automatic |
221 |
activation is enabled (see |
222 |
:ref:`authentication methods policies <auth_methods_policies>`). |
223 |
|
224 |
If all of the above fail to trigger automatic activation, an email is sent to |
225 |
the persons listed in ``HELPDESK``, ``MANAGERS`` and ``ADMINS`` settings, |
226 |
notifing that there is a new user pending for moderation and that it's up to |
227 |
the administrator to decide if the user should be activated. The UI also shows |
228 |
a corresponding 'pending moderation' message to the user. The administrator can |
229 |
activate a user using the ``snf-manage user-modify`` command: |
230 |
|
231 |
.. code-block:: console |
232 |
|
233 |
# command to activate a pending user |
234 |
$ snf-manage user-modify --accept <userid> |
235 |
|
236 |
# command to reject a pending user |
237 |
$ snf-manage user-modify --reject --reject-reason="spammer" <userid> |
238 |
|
239 |
Once the activation process finishes, a greeting message is sent to the user |
240 |
email address and a notification for the activation to the persons listed in |
241 |
``HELPDESK``, ``MANAGERS`` and ``ADMINS`` settings. Once activated the user is |
242 |
able to login and access the Synnefo services. |
243 |
|
244 |
Additional authentication methods |
245 |
````````````````````````````````` |
246 |
|
247 |
Astakos supports third party logins from external identity providers. This |
248 |
can be usefull since it allows users to use their existing credentials to |
249 |
login to astakos service. |
250 |
|
251 |
Currently astakos supports the following identity providers: |
252 |
|
253 |
* `Shibboleth <http://www.internet2.edu/shibboleth>`_ (module name |
254 |
``shibboleth``) |
255 |
* `Google <https://developers.google.com/accounts/docs/OAuth2>`_ (module |
256 |
name ``google``) |
257 |
* `Twitter <https://dev.twitter.com/docs/auth>`_ (module name ``twitter``) |
258 |
* `LinkedIn <http://developer.linkedin.com/documents/authentication>`_ |
259 |
(module name ``linkedin``) |
260 |
|
261 |
To enable any of the above modules (by default only ``local`` accounts are |
262 |
allowed), retrieve and set the required provider settings and append the |
263 |
module name in ``ASTAKOS_IM_MODULES``. |
264 |
|
265 |
.. code-block:: python |
266 |
|
267 |
# settings from https://code.google.com/apis/console/ |
268 |
ASTAKOS_GOOGLE_CLIENT_ID = '1111111111-epi60tvimgha63qqnjo40cljkojcann3.apps.googleusercontent.com' |
269 |
ASTAKOS_GOOGLE_SECRET = 'tNDQqTDKlTf7_LaeUcWTWwZM' |
270 |
|
271 |
# let users signup and login using their google account |
272 |
ASTAKOS_IM_MODULES = ['local', 'google'] |
273 |
|
274 |
|
275 |
.. _auth_methods_policies: |
276 |
|
277 |
Authentication method policies |
278 |
`````````````````````````````` |
279 |
|
280 |
Astakos allows you to override the default policies for each enabled provider |
281 |
separately by adding the approriate settings in your ``.conf`` files in the |
282 |
following format: |
283 |
|
284 |
**ASTAKOS_AUTH_PROVIDER_<module>_<policy>_POLICY** |
285 |
|
286 |
Available policies are: |
287 |
|
288 |
* **CREATE** Users can signup using that provider (default: ``True``) |
289 |
* **REMOVE/ADD** Users can remove/add login method from their profile |
290 |
(default: ``True``) |
291 |
* **AUTOMODERATE** Automatically activate users that signup using that |
292 |
provider (default: ``False``) |
293 |
* **LOGIN** Whether or not users can use the provider to login (default: |
294 |
``True``). |
295 |
|
296 |
e.g. to enable automatic activation for your academic users, while keeping |
297 |
locally signed up users under moderation you can apply the following settings. |
298 |
|
299 |
.. code-block:: python |
300 |
|
301 |
ASTAKOS_AUTH_PROVIDER_SHIBBOLETH_AUTOMODERATE_POLICY = True |
302 |
ASTAKOS_AUTH_PROVIDER_SHIBBOLETH_REMOVE_POLICY = False |
303 |
|
304 |
User login |
305 |
~~~~~~~~~~ |
306 |
|
307 |
During the logging procedure, the user is authenticated by the respective |
308 |
identity provider. |
309 |
|
310 |
If ``ASTAKOS_RECAPTCHA_ENABLED`` is set and the user fails several times |
311 |
(``ASTAKOS_RATELIMIT_RETRIES_ALLOWED`` setting) to provide the correct |
312 |
credentials for a local account, he/she is then prompted to solve a captcha |
313 |
challenge. |
314 |
|
315 |
Upon success, the system renews the token (if it has expired), logins the user |
316 |
and sets the cookie, before redirecting the user to the ``next`` parameter |
317 |
value. |
318 |
|
319 |
Projects and quota |
320 |
~~~~~~~~~~~~~~~~~~ |
321 |
|
322 |
Synnefo supports granting resources and controling their quota through the |
323 |
mechanism of *projects*. A project is considered as a pool of finite |
324 |
resources. Every actual resources allocated by a user (e.g. a Cyclades VM or |
325 |
a Pithos container) is also assigned to a project where the user is a |
326 |
member to. For each resource a project specifies the maximum amount that can |
327 |
be assigned to it and the maximum amount that a single member can assign to it. |
328 |
|
329 |
Default quota |
330 |
````````````` |
331 |
|
332 |
Upon user creation, a special purpose user-specific project is automatically |
333 |
created in order to hold the base quota provided by the system. These *base* |
334 |
projects are identified with the same UUID as the user. |
335 |
|
336 |
To inspect the quota that future users will receive by default through their |
337 |
base projects, check column ``base_default`` in:: |
338 |
|
339 |
# snf-manage resource-list |
340 |
|
341 |
You can modify the default base quota limit for all future users with:: |
342 |
|
343 |
# snf-manage resource-modify <resource_name> --base-default <value> |
344 |
|
345 |
Grant extra quota through projects |
346 |
`````````````````````````````````` |
347 |
|
348 |
A user can apply for a new project through the web interface or the API. |
349 |
Once it is approved by the administrators, the applicant can join the |
350 |
project and let other users in too. |
351 |
|
352 |
A project member can make use of the quota granted by the project by |
353 |
specifying this particular project when creating a new quotable entity. |
354 |
|
355 |
Note that quota are not accumulative: in order to allocate a 100GB disk, |
356 |
one must be in a project that grants at least 100GB; it is not possible to |
357 |
add up quota from different projects. Note also that if allocating an entity |
358 |
requires multiple resources (e.g. cpu and ram for a Cyclades VM) these must |
359 |
be all assigned to a single project. |
360 |
|
361 |
Control projects |
362 |
```````````````` |
363 |
|
364 |
To list pending project applications in astakos:: |
365 |
|
366 |
# snf-manage project-list --pending |
367 |
|
368 |
Note the last column, the application id. To approve it:: |
369 |
|
370 |
# <app id> from the last column of project-list |
371 |
# snf-manage project-control --approve <app id> |
372 |
|
373 |
To deny an application:: |
374 |
|
375 |
# snf-manage project-control --deny <app id> |
376 |
|
377 |
Before taking an action, on can inspect project status, settings and quota |
378 |
limits with:: |
379 |
|
380 |
# snf-manage project-show <project-uuid> |
381 |
|
382 |
For an initialized project, option ``--quota`` also reports the resource |
383 |
usage. |
384 |
|
385 |
Users designated as *project admins* can approve or deny |
386 |
an application through the web interface. In |
387 |
``20-snf-astakos-app-settings.conf`` set:: |
388 |
|
389 |
# UUIDs of users that can approve or deny project applications from the web. |
390 |
ASTAKOS_PROJECT_ADMINS = [<uuid>, ...] |
391 |
|
392 |
Set quota limits |
393 |
```````````````` |
394 |
|
395 |
One can change the quota limits of an initialized project with:: |
396 |
|
397 |
# snf-manage project-modify <project-uuid> --limit <resource_name> <member_limit> <project_limit> |
398 |
|
399 |
One can set base quota for all accepted users (that is, set limits for base |
400 |
project), with possible exceptions, with:: |
401 |
|
402 |
# snf-manage project-modify --all-base-projects --exclude <uuid1>,<uuid2> --limit ... |
403 |
|
404 |
Quota for a given resource are reported for all projects that the user is |
405 |
member in with:: |
406 |
|
407 |
# snf-manage user-show <user-uuid> --quota |
408 |
|
409 |
With option ``--projects``, owned projects and memberships are also reported. |
410 |
|
411 |
Astakos advanced operations |
412 |
--------------------------- |
413 |
|
414 |
Adding "Terms of Use" |
415 |
~~~~~~~~~~~~~~~~~~~~~ |
416 |
|
417 |
Astakos supports versioned terms-of-use. First of all you need to create an |
418 |
html file that will contain your terms. For example, create the file |
419 |
``/usr/share/synnefo/sample-terms.html``, which contains the following: |
420 |
|
421 |
.. code-block:: console |
422 |
|
423 |
<h1>My cloud service terms</h1> |
424 |
|
425 |
These are the example terms for my cloud service |
426 |
|
427 |
Then, add those terms-of-use with the snf-manage command: |
428 |
|
429 |
.. code-block:: console |
430 |
|
431 |
$ snf-manage term-add /usr/share/synnefo/sample-terms.html |
432 |
|
433 |
Your terms have been successfully added and you will see the corresponding link |
434 |
appearing in the Astakos web pages' footer. |
435 |
|
436 |
During the account registration, if there are approval terms, the user is |
437 |
presented with an "I agree with the Terms" checkbox that needs to get checked |
438 |
in order to proceed. |
439 |
|
440 |
In case there are new approval terms that the user has not signed yet, the |
441 |
``signed_terms_required`` view decorator redirects to the ``approval_terms`` |
442 |
view, so the user will be presented with the new terms the next time he/she |
443 |
logins. |
444 |
|
445 |
Enabling reCAPTCHA |
446 |
~~~~~~~~~~~~~~~~~~ |
447 |
|
448 |
Astakos supports the `reCAPTCHA <http://www.google.com/recaptcha>`_ feature. |
449 |
If enabled, it protects the Astakos forms from bots. To enable the feature, go |
450 |
to https://www.google.com/recaptcha/admin/create and create your own reCAPTCHA |
451 |
key pair. Then edit ``/etc/synnefo/20-snf-astakos-app-settings.conf`` and set |
452 |
the corresponding variables to reflect your newly created key pair. Finally, set |
453 |
the ``ASTAKOS_RECAPTCHA_ENABLED`` variable to ``True``: |
454 |
|
455 |
.. code-block:: console |
456 |
|
457 |
ASTAKOS_RECAPTCHA_PUBLIC_KEY = 'example_recaptcha_public_key!@#$%^&*(' |
458 |
ASTAKOS_RECAPTCHA_PRIVATE_KEY = 'example_recaptcha_private_key!@#$%^&*(' |
459 |
|
460 |
ASTAKOS_RECAPTCHA_ENABLED = True |
461 |
|
462 |
Restart the service on the Astakos node(s) and you are ready: |
463 |
|
464 |
.. code-block:: console |
465 |
|
466 |
# /etc/init.d/gunicorn restart |
467 |
|
468 |
Checkout your new Sign up page. If you see the reCAPTCHA box, you have setup |
469 |
everything correctly. |
470 |
|
471 |
|
472 |
Astakos internals |
473 |
----------------- |
474 |
|
475 |
X-Auth-Token |
476 |
~~~~~~~~~~~~ |
477 |
|
478 |
Alice requests a specific resource from a cloud service e.g.: Pithos. In the |
479 |
request she supplies the `X-Auth-Token` to identify whether she is eligible to |
480 |
perform the specific task. The service contacts Astakos through its |
481 |
``/account/v1.0/authenticate`` api call (see :ref:`authenticate-api-label`) |
482 |
providing the specific ``X-Auth-Token``. Astakos checkes whether the token |
483 |
belongs to an active user and it has not expired and returns a dictionary |
484 |
containing user related information. Finally the service uses the ``uniq`` |
485 |
field included in the dictionary as the account string to identify the user |
486 |
accessible resources. |
487 |
|
488 |
.. _authentication-label: |
489 |
|
490 |
Django Auth methods and Backends |
491 |
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
492 |
|
493 |
Astakos incorporates Django user authentication system and extends its User model. |
494 |
|
495 |
Since username field of django User model has a limitation of 30 characters, |
496 |
AstakosUser is **uniquely** identified by the ``email`` instead. Therefore, |
497 |
``astakos.im.authentication_backends.EmailBackend`` is served to authenticate a |
498 |
user using email if the first argument is actually an email, otherwise tries |
499 |
the username. |
500 |
|
501 |
A new AstakosUser instance is assigned with a uui as username and also with a |
502 |
``auth_token`` used by the cloud services to authenticate the user. |
503 |
``astakos.im.authentication_backends.TokenBackend`` is also specified in order |
504 |
to authenticate the user using the email and the token fields. |
505 |
|
506 |
Logged on users can perform a number of actions: |
507 |
|
508 |
* access and edit their profile via: ``/im/profile``. |
509 |
* change their password via: ``/im/password`` |
510 |
* send feedback for grnet services via: ``/im/send_feedback`` |
511 |
* logout (and delete cookie) via: ``/im/logout`` |
512 |
|
513 |
Internal Astakos requests are handled using cookie-based Django user sessions. |
514 |
|
515 |
External systems should forward to the ``/login`` URI. The server, |
516 |
depending on its configuration will redirect to the appropriate login page. |
517 |
When done with logging in, the service's login URI should redirect to the URI |
518 |
provided with next, adding user and token parameters, which contain the email |
519 |
and token fields respectively. |
520 |
|
521 |
The login URI accepts the following parameters: |
522 |
|
523 |
====================== ========================= |
524 |
Request Parameter Name Value |
525 |
====================== ========================= |
526 |
next The URI to redirect to when the process is finished |
527 |
renew Force token renewal (no value parameter) |
528 |
force Force logout current user (no value parameter) |
529 |
====================== ========================= |
530 |
|
531 |
External systems inside the ``ASTAKOS_COOKIE_DOMAIN`` scope can acquire the |
532 |
user information by the cookie identified by ``ASTAKOS_COOKIE_NAME`` setting |
533 |
(set during the login procedure). |
534 |
|
535 |
Finally, backend systems having acquired a token can use the |
536 |
:ref:`authenticate-api-label` API call from a private network or through HTTPS. |
537 |
|
538 |
|
539 |
File/Object Storage Service (Pithos+) |
540 |
==================================== |
541 |
|
542 |
Pithos+ is the Synnefo component that implements a storage service and exposes |
543 |
the associated OpenStack REST APIs with custom extensions. |
544 |
|
545 |
Pithos+ advanced operations |
546 |
--------------------------- |
547 |
|
548 |
Enable separate domain for serving user content |
549 |
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
550 |
|
551 |
Since Synnefo v0.15, there is a possibility to serve untrusted user content |
552 |
in an isolated domain. |
553 |
|
554 |
Enabling this feature consists of the following steps: |
555 |
|
556 |
#. **Declare new domain in apache server** |
557 |
|
558 |
In order to enable the apache server to serve several domains it is required |
559 |
to setup several virtual hosts. |
560 |
Therefore, for adding the new domain e.g. "user-content.example.com", append |
561 |
the following in ``/etc/apache2/sites-available/synnefo-ssl``: |
562 |
|
563 |
.. code-block:: console |
564 |
|
565 |
<VirtualHost _default_:443> |
566 |
ServerName user-content.example.com |
567 |
|
568 |
Alias /static "/usr/share/synnefo/static" |
569 |
|
570 |
# SetEnv no-gzip |
571 |
# SetEnv dont-vary |
572 |
|
573 |
AllowEncodedSlashes On |
574 |
|
575 |
RequestHeader set X-Forwarded-Protocol "https" |
576 |
|
577 |
<Proxy * > |
578 |
Order allow,deny |
579 |
Allow from all |
580 |
</Proxy> |
581 |
|
582 |
SetEnv proxy-sendchunked |
583 |
SSLProxyEngine off |
584 |
ProxyErrorOverride off |
585 |
|
586 |
ProxyPass /static ! |
587 |
ProxyPass / http://localhost:8080/ retry=0 |
588 |
ProxyPassReverse / http://localhost:8080/ |
589 |
|
590 |
RewriteEngine On |
591 |
RewriteCond %{THE_REQUEST} ^.*(\\r|\\n|%0A|%0D).* [NC] |
592 |
RewriteRule ^(.*)$ - [F,L] |
593 |
|
594 |
SSLEngine on |
595 |
SSLCertificateFile /etc/ssl/certs/ssl-cert-snakeoil.pem |
596 |
SSLCertificateKeyFile /etc/ssl/private/ssl-cert-snakeoil.key |
597 |
</VirtualHost> |
598 |
|
599 |
.. note:: Consider also to purchase and install a certificate for the new |
600 |
domain. |
601 |
|
602 |
|
603 |
Finally, restart the apache server:: |
604 |
|
605 |
pithos-host$ /etc/init.d/apache2 restart |
606 |
|
607 |
#. **Register Pithos+ as an OAuth2 client in Astakos** |
608 |
|
609 |
Starting from synnefo version 0.15, in order to view the content of a |
610 |
protected resource, Pithos+ (on behalf of the user) has to be granted |
611 |
authorization for the specific resource by Astakos. |
612 |
|
613 |
During the authorization grant procedure, Pithos+ has to authenticate |
614 |
itself with Astakos since the latter has to prevent serving requests by |
615 |
unknown/unauthorized clients. |
616 |
|
617 |
Therefore, in the installation guide you were guided to register Pithos+ |
618 |
as an OAuth2 client in Astakos. |
619 |
|
620 |
.. note:: You can see the registered clients by running:: |
621 |
astakos-host$ snf-manage oauth2-client-list -o identifier,redirect_urls,is_trusted |
622 |
|
623 |
However, requests originated from the new domain will be rejected since |
624 |
Astakos is ignorant about the new domain. |
625 |
|
626 |
Therefore, you need to register a new client pointing to the unsafe domain. |
627 |
To do so, use the following command:: |
628 |
|
629 |
astakos-host$ snf-manage oauth2-client-add pithos-unsafe-domain --secret=<secret> --is-trusted --url https://user-content.example.com/pithos/ui/view |
630 |
|
631 |
|
632 |
.. note:: You can also unregister the client pointing to the safe domain, |
633 |
since it will no longer be useful. |
634 |
To do so, run the following:: |
635 |
|
636 |
astakos-host$ snf-manage oauth2-client-remove pithos-view |
637 |
|
638 |
#. **Update Pithos+ configuration** |
639 |
|
640 |
Respectively, the ``PITHOS_OAUTH2_CLIENT_CREDENTIALS`` setting should be |
641 |
updated to contain the credentials of the client registered in the previous |
642 |
step. |
643 |
|
644 |
Furthermore, you need to restrict all the requests for user content |
645 |
to be served exclusively by the unsafe domain. |
646 |
|
647 |
To enable this, set the ``PITHOS_UNSAFE_DOMAIN`` setting to the value |
648 |
of the new domain e.g. "user-content.example.com" |
649 |
|
650 |
Finally, restart the gunicorn server:: |
651 |
|
652 |
pithos-host$ /etc/init.d/gunicorn restart |
653 |
|
654 |
|
655 |
Compute/Network/Image Service (Cyclades) |
656 |
======================================== |
657 |
|
658 |
Introduction |
659 |
------------ |
660 |
|
661 |
Cyclades is the Synnefo component that implements Compute, Network and Image |
662 |
services and exposes the associated OpenStack REST APIs. By running Cyclades |
663 |
you can provide a cloud that can handle thousands of virtual servers and |
664 |
networks. |
665 |
|
666 |
Cyclades does not include any virtualization software and knows nothing about |
667 |
the low-level VM management operations, e.g. handling of VM creation or |
668 |
migrations among physical nodes. Instead, Cyclades is the component that |
669 |
handles multiple Ganeti backends and exposes the REST APIs. The administrator |
670 |
can expand the infrastructure dynamically either by adding more Ganeti nodes |
671 |
or by adding new Ganeti clusters. Cyclades issue VM control commands to Ganeti |
672 |
via Ganeti's remote API and receive asynchronous notifications from Ganeti |
673 |
backends whenever the state of a VM changes, due to Synnefo- or |
674 |
administrator-initiated commands. |
675 |
|
676 |
Cyclades is the action orchestrator and the API layer on top of multiple Ganeti |
677 |
clusters. By this decoupled design, Ganeti cluster are self-contained and |
678 |
the administrator has complete control on them without Cyclades knowing about |
679 |
it. For example a VM migration to a different physical node is transparent |
680 |
to Cyclades. |
681 |
|
682 |
Working with Cyclades |
683 |
--------------------- |
684 |
|
685 |
Flavors |
686 |
~~~~~~~ |
687 |
|
688 |
When creating a VM, the user must specify the `flavor` of the virtual server. |
689 |
Flavors are the virtual hardware templates, and provide a description about |
690 |
the number of CPUs, the amount of RAM, and the size of the disk of the VM. |
691 |
Besides the size of the disk, Cyclades flavors describe the storage backend |
692 |
that will be used for the virtual server. |
693 |
|
694 |
Flavors are created by the administrator and the user can select one of the |
695 |
available flavors. After VM creation, the user can resize his VM, by |
696 |
adding/removing CPU and RAM. |
697 |
|
698 |
Cyclades support different storage backends that are described by the disk |
699 |
template of the flavor, which is mapped to Ganeti's instance `disk template`. |
700 |
Currently the available disk templates are the following: |
701 |
|
702 |
* `file`: regulars file |
703 |
* `sharedfile`: regular files on a shared directory, e.g. NFS |
704 |
* `plain`: logical volumes |
705 |
* `drbd`: drbd on top of lvm volumes |
706 |
* `rbd`: rbd volumes residing inside a RADOS cluster |
707 |
* `ext`: disks provided by an external shared storage. |
708 |
|
709 |
- `ext_archipelago`: External shared storage provided by |
710 |
`Archipelago <http://www.synnefo.org/docs/archipelago/latest/index.html>`_. |
711 |
|
712 |
Flavors are created by the administrator using `snf-manage flavor-create` |
713 |
command. The command takes as argument number of CPUs, amount of RAM, the size |
714 |
of the disks and the disk templates and create the flavors that belong to the |
715 |
cartesian product of the specified arguments. For example, the following |
716 |
command will create two flavors of `40G` disk size with `drbd` disk template, |
717 |
`4G` RAM and `2` or `4` CPUs. |
718 |
|
719 |
.. code-block:: console |
720 |
|
721 |
$ snf-manage flavor-create 2,4 4096 40 drbd |
722 |
|
723 |
To see the available flavors, run `snf-manage flavor-list` command. The |
724 |
administrator can delete a flavor by using `flavor-modify` command: |
725 |
|
726 |
.. code-block:: console |
727 |
|
728 |
$ snf-manage flavor-modify --deleted=True <flavor_id> |
729 |
|
730 |
Finally, the administrator can set if new servers can be created from a flavor |
731 |
or not, by setting the `allow_create` attribute: |
732 |
|
733 |
.. code-block:: console |
734 |
|
735 |
$ snf-manage flavor-modify --allow-create=False <flavor_id> |
736 |
|
737 |
Flavors that are marked with `allow_create=False` cannot be used by users to |
738 |
create new servers. However, they can still be used to resize existing VMs. |
739 |
|
740 |
|
741 |
Images |
742 |
~~~~~~ |
743 |
|
744 |
When creating a VM the user must also specify the `image` of the virtual |
745 |
server. Images are the static templates from which VM instances are |
746 |
initiated. Cyclades uses Pithos to store system and user-provided images, |
747 |
taking advantage of all Pithos features, like deduplication and syncing |
748 |
protocol. An image is a file stored to Pithos with additional metadata that |
749 |
are describing the image, e.g. the OS family or the root partition. To create |
750 |
a new image, the administrator or the user has to upload it a file to Pithos, |
751 |
and then register it as an Image with Cyclades. Then the user can use this |
752 |
image to spawn new VMs from it. |
753 |
|
754 |
Images can be private, public or shared between users, exactly like Pithos |
755 |
files. Since user-provided public images can be untrusted, the administrator |
756 |
can denote which users are trusted by adding them to the |
757 |
``UI_SYSTEM_IMAGES_OWNERS`` setting in the |
758 |
`/etc/synnefo/20-snf-cyclades-app-ui.conf` file. Images of those users are |
759 |
properly displayed in the UI. |
760 |
|
761 |
When creating a new VM, Cyclades pass the location of the image and it's |
762 |
metadata to Ganeti. After Ganeti creates the instance's disk, `snf-image` |
763 |
will copy the image to the new disk and perform the image customization |
764 |
phase. During the phase, `snf-image` sends notifications to Cyclades about |
765 |
the progress of the image deployment and customization. Customization includes |
766 |
resizing the root file system, file injection (e.g. SSH keys) and setting |
767 |
a custom hostname. For better understanding of `snf-image` read the |
768 |
corresponding `documentation |
769 |
<http://www.synnefo.org/docs/snf-image/latest/index.html>`_. |
770 |
|
771 |
For passing sensitive data about the image to Ganeti, like the VMs password, |
772 |
Cyclades keeps all sensitive data in memory caches (memcache) and never allows |
773 |
them to hit the disk. The data are exposed to `snf-image` via an one-time URL |
774 |
that is exposed from the `vmapi` application. So, instead of passing sensitive |
775 |
data to `snf-image` via Ganeti, Cyclades pass an one-time configuration URL |
776 |
that contains a random UUID. After `snf-image` gets the sensitive data, the |
777 |
URL is invalidated so no one else can access them. |
778 |
|
779 |
The administrator can register images, exactly like users, using a system user |
780 |
(a user that is defined in the ``UI_SYSTEM_IMAGES_OWNERS`` setting). For |
781 |
example, the following command will register the |
782 |
`pithos://u53r-un1qu3-1d/images/debian_base-6.0-7-x86_64.diskdump` as an |
783 |
image to Cyclades: |
784 |
|
785 |
.. code-block:: console |
786 |
|
787 |
$ kamaki image register --name="Debian Base" \ |
788 |
--location=pithos://u53r-un1qu3-1d/images/debian_base-6.0-7-x86_64.diskdump \ |
789 |
--public \ |
790 |
--disk-format=diskdump \ |
791 |
--property OSFAMILY=linux --property ROOT_PARTITION=1 \ |
792 |
--property description="Debian Squeeze Base System" \ |
793 |
--property size=451 --property kernel=2.6.32 --property GUI="No GUI" \ |
794 |
--property sortorder=1 --property USERS=root --property OS=debian |
795 |
|
796 |
Deletion of an image is done via `kamaki image unregister` command, which will |
797 |
delete the Cyclades Images but will leave the Pithos file as is (unregister). |
798 |
|
799 |
Apart from using `kamaki` to see and hangle the available images, the |
800 |
administrator can use `snf-manage image-list` and `snf-manage image-show` |
801 |
commands to list and inspect the available public images. Also, the `--user-id` |
802 |
option can be used the see the images of a specific user. |
803 |
|
804 |
Virtual Servers |
805 |
~~~~~~~~~~~~~~~ |
806 |
|
807 |
As mentioned, Cyclades uses Ganeti for management of VMs. The administrator can |
808 |
handle Cyclades VMs just like any other Ganeti instance, via `gnt-instance` |
809 |
commands. All Ganeti instances that belong to Synnefo, are separated from |
810 |
others, by a prefix in their names. This prefix is defined in |
811 |
``BACKEND_PREFIX_ID`` setting in |
812 |
``/etc/synnefo/20-snf-cyclades-app-backend.conf``. |
813 |
|
814 |
Apart from handling Cyclades VM at the Ganeti level, the administrator can |
815 |
also use the `snf-manage server-*` commands. These command cover the most |
816 |
common tasks that are relative with VM handling. Below we describe come |
817 |
of them, but for more information you can use the `--help` option of all |
818 |
`snf-manage server-* commands`. These command cover the most |
819 |
|
820 |
The `snf-manage server-create` command can be used to create a new VM for some |
821 |
user. This command can be useful when the administrator wants to test Cyclades |
822 |
functionality without starting the API service, e.g. after an upgrade. Also, by |
823 |
using `--backend-id` option, the VM will be created in the specified backend, |
824 |
bypassing automatic VM allocation. |
825 |
|
826 |
.. code-block:: console |
827 |
|
828 |
$ snf-manage server-create --flavor-id=1 --image-id=fc0f6858-f962-42ce-bf9a-1345f89b3d5e \ |
829 |
--user-id=7cf4d078-67bf-424d-8ff2-8669eb4841ea --backend-id=2 \ |
830 |
--password='example_passw0rd' --name='test_vm' |
831 |
|
832 |
The above commnd will create a new VM for user |
833 |
`7cf4d078-67bf-424d-8ff2-8669eb4841ea` in the Ganeti backend with ID 2. By |
834 |
default this command will issue a Ganeti job to create the VM |
835 |
(`OP_INSTANCE_CREATE`) and return. As in other commands, the `--wait=True` |
836 |
option can be used in order to wait for the successful completion of the job. |
837 |
|
838 |
`snf-manage server-list` command can be used to list all the available servers. |
839 |
The command supports some useful options, like listing servers of a user, |
840 |
listing servers that exist in a Ganeti backend and listing deleted servers. |
841 |
Also, as in most of `*-list` commands, the `--filter-by` option can be used to |
842 |
filter the results. For example, the following command will only display the |
843 |
started servers of a specific flavor: |
844 |
|
845 |
.. code-block:: console |
846 |
|
847 |
$ snf-manage server-list --filter-by="operstate=STARTED,flavor=<flavor_id>" |
848 |
|
849 |
Another very useful command is the `server-inspect` command which will display |
850 |
all available information about the state of the server in DB and the state |
851 |
of the server in the Ganeti backend. The output will give you an easy overview |
852 |
about the state of the VM which can be useful for debugging. |
853 |
|
854 |
Also the administrator can `suspend` a user's VM, using the `server-modify` |
855 |
command: |
856 |
|
857 |
.. code-block:: console |
858 |
|
859 |
$ snf-manage server-modify --suspended=True <server_id> |
860 |
|
861 |
The user is forbidden to do any action on an administratively suspended VM, |
862 |
which is useful for abuse cases. |
863 |
|
864 |
Ganeti backends |
865 |
~~~~~~~~~~~~~~~ |
866 |
|
867 |
Since v0.11, Synnefo is able to manage multiple Ganeti clusters (backends) |
868 |
making it capable to scale linearly to tens of thousands of VMs. Backends |
869 |
can be dynamically added or removed via `snf-manage` commands. |
870 |
|
871 |
Each newly created VM is allocated to a Ganeti backend by the Cyclades backend |
872 |
allocator. The VM is "pinned" to this backend, and can not change through its |
873 |
lifetime. The backend allocator decides in which backend to spawn the VM based |
874 |
on the available resources of each backend, trying to balance the load between |
875 |
them. Also, Networks are created to all Ganeti backends, in order to ensure |
876 |
that VMs residing on different backends can be connected to the same networks. |
877 |
|
878 |
A backend can be marked as `drained` in order to be excluded from automatic |
879 |
servers allocation and not receive new servers. Also, a backend can be marked |
880 |
as `offline` to denote that the backend is not healthy (e.g. broken master) |
881 |
and avoid the penalty of connection timeouts. |
882 |
|
883 |
Finally, Cyclades is able to manage Ganeti backends with different enabled |
884 |
hypervisors (`kvm`, `xen`), and different enabled disk templates. |
885 |
|
886 |
Listing existing backends |
887 |
````````````````````````` |
888 |
To list all the Ganeti backends known to Synnefo, we run: |
889 |
|
890 |
.. code-block:: console |
891 |
|
892 |
$ snf-manage backend-list |
893 |
|
894 |
Adding a new Ganeti backend |
895 |
``````````````````````````` |
896 |
Backends are dynamically added under the control of Synnefo with `snf-manage |
897 |
backend-add` command. In this section it is assumed that a Ganeti cluster, |
898 |
named ``cluster.example.com`` is already up and running and configured to be |
899 |
able to host Synnefo VMs. |
900 |
|
901 |
To add this Ganeti cluster, we run: |
902 |
|
903 |
.. code-block:: console |
904 |
|
905 |
$ snf-manage backend-add --clustername=cluster.example.com --user="synnefo_user" --pass="synnefo_pass" |
906 |
|
907 |
where ``clustername`` is the Cluster hostname of the Ganeti cluster, and |
908 |
``user`` and ``pass`` are the credentials for the `Ganeti RAPI user |
909 |
<http://docs.ganeti.org/ganeti/2.8/html/rapi.html#users-and-passwords>`_. All |
910 |
backend attributes can be also changed dynamically using the `snf-manage |
911 |
backend-modify` command. |
912 |
|
913 |
``snf-manage backend-add`` will also create all existing public networks to |
914 |
the new backend. You can verify that the backend is added, by running |
915 |
`snf-manage backend-list`. |
916 |
|
917 |
Note that no VMs will be spawned to this backend, since by default it is in a |
918 |
``drained`` state after addition in order to manually verify the state of the |
919 |
backend. |
920 |
|
921 |
So, after making sure everything works as expected, make the new backend active |
922 |
by un-setting the ``drained`` flag. You can do this by running: |
923 |
|
924 |
.. code-block:: console |
925 |
|
926 |
$ snf-manage backend-modify --drained=False <backend_id> |
927 |
|
928 |
Allocation of VMs in Ganeti backends |
929 |
```````````````````````````````````` |
930 |
As already mentioned, the Cyclades backend allocator is responsible for |
931 |
allocating new VMs to backends. This allocator does not choose the exact Ganeti |
932 |
node that will host the VM but just the Ganeti backend. The exact node is |
933 |
chosen by the Ganeti cluster's allocator (hail). |
934 |
|
935 |
The decision about which backend will host a VM is based on the available |
936 |
resources. The allocator computes a score for each backend, that shows its load |
937 |
factor, and the one with the minimum score is chosen. The admin can exclude |
938 |
backends from the allocation phase by marking them as ``drained`` by running: |
939 |
|
940 |
.. code-block:: console |
941 |
|
942 |
$ snf-manage backend-modify --drained=True <backend_id> |
943 |
|
944 |
The backend resources are periodically updated, at a period defined by |
945 |
the ``BACKEND_REFRESH_MIN`` setting, or by running `snf-manage |
946 |
backend-update-status` command. It is advised to have a cron job running this |
947 |
command at a smaller interval than ``BACKEND_REFRESH_MIN`` in order to remove |
948 |
the load of refreshing the backends stats from the VM creation phase. |
949 |
|
950 |
Finally, the admin can decide to have a user's VMs being allocated to a |
951 |
specific backend, with the ``BACKEND_PER_USER`` setting. This is a mapping |
952 |
between users and backends. If the user is found in ``BACKEND_PER_USER``, then |
953 |
Synnefo allocates all his/hers VMs to the specific backend in the variable, |
954 |
even if is marked as drained (useful for testing). |
955 |
|
956 |
.. _alloc_disk_templates: |
957 |
|
958 |
Allocation based on disk-templates |
959 |
********************************** |
960 |
|
961 |
Besides the available resources of each Ganeti backend, the allocator takes |
962 |
into consideration the disk template of the instance when trying to allocate it |
963 |
to a Ganeti backend. Specifically, the allocator checks if the flavor of the |
964 |
instance belongs to the available disk templates of each Ganeti backend. |
965 |
|
966 |
A Ganeti cluster has a list of enabled disk templates |
967 |
(`--enabled-disk-templates`) and a list of allowed disk templates for new |
968 |
instances (`--ipolicy-disk-templates`). See the `gnt-cluster` manpage for more |
969 |
details about these options. |
970 |
|
971 |
When Synnefo allocates an instance, it checks whether the disk template of the |
972 |
new instance belongs both in the enabled and ipolicy disk templates. You can |
973 |
see the list of the available disk-templates by running `snf-manage |
974 |
backend-list`. This list should be updated automatically after changing |
975 |
these options in Ganeti and it can also be updated by running `snf-manage |
976 |
backend-update-status`. |
977 |
|
978 |
So the administrator, can route instances on different backends based on their |
979 |
flavor disk template, by modifying the enabled or ipolicy disk templates of |
980 |
each backend. Also, the administrator can route instances between different |
981 |
nodes of the same Ganeti backend, by modifying the same options at the |
982 |
nodegroup level (see `gnt-group` manpage for mor details). |
983 |
|
984 |
Removing an existing Ganeti backend |
985 |
``````````````````````````````````` |
986 |
In order to remove an existing backend from Synnefo, you must first make |
987 |
sure that there are not active servers in the backend, and then run: |
988 |
|
989 |
.. code-block:: console |
990 |
|
991 |
$ snf-manage backend-remove <backend_id> |
992 |
|
993 |
|
994 |
Virtual Networks |
995 |
~~~~~~~~~~~~~~~~ |
996 |
|
997 |
Cyclades also implements the Network service and exposes the Quantum Openstack |
998 |
API. Cyclades supports full IPv4 and IPv6 connectivity to the public internet |
999 |
for it's VMs. Also, Cyclades provides L2 and L3 virtual private networks, |
1000 |
giving the user freedom to create arbitraty network topologies of |
1001 |
interconnected VMs. |
1002 |
|
1003 |
Public networking is desployment specific and must be customized based on the |
1004 |
specific needs of the system administrator. Private virtual networks can be |
1005 |
provided by different network technologies which are exposed as different |
1006 |
network flavors. For better understanding of networking please refer to the |
1007 |
:ref:`Network <networks>` section. |
1008 |
|
1009 |
A Cyclades virtual network is an isolated Layer-2 broadcast domain. A network |
1010 |
can also have an associated IPv4 and IPv6 subnet representing the Layer-3 |
1011 |
characteristics of the network. Each subnet represents an IP address block |
1012 |
that is used in order to assign addresses to VMs. |
1013 |
|
1014 |
To connect a VM to a network, a port must be created, which represent a virtual |
1015 |
port on a network switch. VMs are connected to networks by attaching a virtual |
1016 |
interface to a port. |
1017 |
|
1018 |
Cyclades also supports `floating IPs`, which are public IPv4 addresses that |
1019 |
can dynamically(hotplug-able) be added and removed to VMs. Floating IPs are |
1020 |
a quotable resource that is allocated to each user. Unlike other cloud |
1021 |
platforms, floating IPs are not implemented using 1-1 NAT to a ports private |
1022 |
IP. Instead, floating IPs are directly assigned to virtual interfaces of VMs. |
1023 |
|
1024 |
Exactly like VMS, networks can be handled as Ganeti networks via `gnt-network` |
1025 |
commands. All Ganeti networks that belong to Synnefo are named with the prefix |
1026 |
`${BACKEND_PREFIX_ID}-net-`. Also, there are a number of `snf-manage` commands |
1027 |
for handling of `networks`, `subnets`, `ports` and `floating IPs`. Below |
1028 |
we will present a use case scenario using some of these commands. For better |
1029 |
understanding of these commands, refer to their help messages. |
1030 |
|
1031 |
Create a virtual private network for user |
1032 |
`7cf4d078-67bf-424d-8ff2-8669eb4841ea` using the `PHYSICAL_VLAN` flavor, which |
1033 |
means that the network will be uniquely assigned a phsyical VLAN. The network |
1034 |
is assigned an IPv4 subnet, described by it's CIDR and gateway. Also, |
1035 |
the `--dhcp=True` option is used, to make `nfdhcpd` response to DHCP queries |
1036 |
from VMs. |
1037 |
|
1038 |
.. code-block:: console |
1039 |
|
1040 |
$ snf-manage network-create --owner=7cf4d078-67bf-424d-8ff2-8669eb4841ea --name=prv_net-1 \ |
1041 |
--subnet=192.168.2.0/24 --gateway=192.168.2.1 --dhcp=True --flavor=PHYSICAL_VLAN |
1042 |
|
1043 |
Inspect the state of the network in Cyclades DB and in all the Ganeti backends: |
1044 |
|
1045 |
.. code-block:: console |
1046 |
|
1047 |
$ snf-manage network-inspect <network_id> |
1048 |
|
1049 |
Inspect the state of the network's subnet, containg an overview of the |
1050 |
subnet's IPv4 address allocation pool: |
1051 |
|
1052 |
.. code-block:: console |
1053 |
|
1054 |
$ snf-manage subnet-inspect <subnet_id> |
1055 |
|
1056 |
Connect a VM to the created private network. The port will be automatically |
1057 |
be assigned an IPv4 address from one of the network's available IPs. This |
1058 |
command will result in sending an `OP_INSTANCE_MODIFY` Ganeti command and |
1059 |
attaching a NIC to the specified Ganeti instance. |
1060 |
|
1061 |
.. code-block:: console |
1062 |
|
1063 |
$ snf-manage port-create --network=<network_id> --server=<server_id> |
1064 |
|
1065 |
Inspect the state of the the port in Cyclades DB and in the Ganeti backend: |
1066 |
|
1067 |
.. code-block:: console |
1068 |
|
1069 |
$ snf-manage port-inspect <port_id> |
1070 |
|
1071 |
Disconnect the VM from the network and delete the network: |
1072 |
|
1073 |
.. code-block:: console |
1074 |
|
1075 |
$ snf-manage port-remove <port_id> |
1076 |
$ snf-manage network-remove <network_id> |
1077 |
|
1078 |
|
1079 |
Enabling DHCP |
1080 |
````````````` |
1081 |
|
1082 |
When connecting a VM to a network, Cyclades will automatically assign an IPv4 |
1083 |
address from the IPv4 or/and IPv6 subnets of the network. If the network has |
1084 |
no subnets, then it will not be assigned any IP address. |
1085 |
|
1086 |
If the network has DHCP enabled, then `nfdhcpd` daemon, which must be running |
1087 |
on all Ganeti nodes, will respond to DHCP queries from VMs and assign to them |
1088 |
the IP address that was allocated by Cyclades. DCHP can be enabled/disabled |
1089 |
using the `--dhcp` option of `network-create` command. |
1090 |
|
1091 |
|
1092 |
Public network connectivity |
1093 |
``````````````````````````` |
1094 |
|
1095 |
Since v0.14, users are able to dynamically connect and disconnect their VMs |
1096 |
from public networks. In order to do that, they have to use a `floating IP`. |
1097 |
Floating IPs are basically public IPv4 addresses that can be dynamically |
1098 |
attached and detached from VMs. The user creates a floating IP address from a |
1099 |
network that has set the `floating_ip_pool` attribute. The floating IP is |
1100 |
accounted to the user, who can then connect his VMs to public networks by |
1101 |
creating ports that they are using this floating IP. Performing this work-flow |
1102 |
from `snf-manage` would look like this: |
1103 |
|
1104 |
.. code-block:: console |
1105 |
|
1106 |
$ snf-manage network-list --filter-by="floating_ip_pool=True" |
1107 |
id name user.uuid state public subnet.ipv4 gateway.ipv4 drained floating_ip_pool |
1108 |
--------------------------------------------------------------------------------------------- |
1109 |
1 Internet None ACTIVE True 10.2.1.0/24 10.2.1.1 False True |
1110 |
|
1111 |
$ snf-manage floating-ip-create --owner=7cf4d078-67bf-424d-8ff2-8669eb4841ea --network=1 |
1112 |
|
1113 |
$ snf-manage floating-ip-list --user=7cf4d078-67bf-424d-8ff2-8669eb4841ea |
1114 |
id address network user.uuid server |
1115 |
------------------------------------------------------------------------ |
1116 |
38 10.2.1.2 1 7cf4d078-67bf-424d-8ff2-8669eb4841ea 42 |
1117 |
|
1118 |
$ snf-manage port-create --owner=7cf4d078-67bf-424d-8ff2-8669eb4841ea --network=1 \ |
1119 |
--ipv4-address=10.2.1.2 --floating-ip=38 |
1120 |
|
1121 |
$ snf-manage port-list --user=7cf4d078-67bf-424d-8ff2-8669eb4841ea |
1122 |
id user.uuid mac_address network server_id fixed_ips state |
1123 |
-------------------------------------------------------------------------------------------------- |
1124 |
163 7cf4d078-67bf-424d-8ff2-8669eb4841ea aa:00:00:45:13:98 1 77 10.2.1.2 ACTIVE |
1125 |
|
1126 |
$ snf-manage port-remove 163 |
1127 |
$ snf-manage floating-ip-remove 38 |
1128 |
|
1129 |
Users do not have permission to connect and disconnect VMs from public |
1130 |
networks without using a floating IP address. However, the administrator |
1131 |
have the ability to perform this tasks, using `port-create` and `port-remove` |
1132 |
commands. |
1133 |
|
1134 |
Network connectivity for newly created servers |
1135 |
`````````````````````````````````````````````` |
1136 |
|
1137 |
When creating a virtual server, the user can specify the networks that the |
1138 |
newly created server will be connected to. Beyond this, the administrator can |
1139 |
define a list of networks that every new server will be forced to connect to. |
1140 |
For example, you can enforce all VMs to be connected to a public network |
1141 |
containing a metadata server. The networks must be specified in the |
1142 |
``CYCLADES_FORCED_SERVER_NETWORKS`` that exists in the |
1143 |
``/etc/synnefo/20-snf-cyclades-app-api.conf``. For the networks in this |
1144 |
setting, no access control or quota policy are enforced! |
1145 |
|
1146 |
Finally, the administrator can define a list of networks that new servers will |
1147 |
be connected, *if the user has not* specified networks in the request to create |
1148 |
the server. Access control and quota policy are enforced, just as if the user |
1149 |
had specified these networks. The list of these networks is defined in the |
1150 |
``CYCLADES_DEFAULT_SERVER_NETWORKS`` that exists in the |
1151 |
``/etc/synnefo/20-snf-cyclades-app-api.conf``. This setting should only |
1152 |
be used if Cyclades are being accessed by external clients that are |
1153 |
unaware of the `Neutron API extensions` in the `Compute API`. |
1154 |
|
1155 |
Each member of the above mentioned settings can be: |
1156 |
|
1157 |
* a network UUID |
1158 |
* a tuple of network UUIDs: the server will be connected to only one of these |
1159 |
networks, e.g. one that has a free IPv4 address |
1160 |
* `SNF:ANY_PUBLIC_IPV4`: the server will be connected to any network with |
1161 |
an IPv4 subnet defined |
1162 |
* `SNF:ANY_PUBLIC_IPV6`: the server will be connected to any network with |
1163 |
only an IPv6 subnet defined. |
1164 |
* `SNF:ANY_PUBLIC`: the server will be connected to any public network. |
1165 |
|
1166 |
Public IP accounting |
1167 |
```````````````````` |
1168 |
|
1169 |
There are many use cases, e.g. abuse ports, where you need to find which user |
1170 |
or which server had a public IP address. For this reason, Cyclades keeps track |
1171 |
usage of public IPv4/IPv6 addresses. Specifically, it keeps the date and time |
1172 |
that each public IP address was allocated and released from a virtual server. |
1173 |
This information can be found using `ip-list` command: |
1174 |
|
1175 |
.. code-block:: console |
1176 |
|
1177 |
$ snf-manage ip-list |
1178 |
|
1179 |
Show usage of a specific address: |
1180 |
$ snf-manage ip-list --address=192.168.2.1 |
1181 |
|
1182 |
Show public IPs of a specific server: |
1183 |
$ snf-manage ip-list --server=<server_id> |
1184 |
|
1185 |
|
1186 |
Managing Network Resources |
1187 |
`````````````````````````` |
1188 |
|
1189 |
Proper operation of the Cyclades Network Service depends on the unique |
1190 |
assignment of specific resources to each type of virtual network. Specifically, |
1191 |
these resources are: |
1192 |
|
1193 |
* IP addresses. Cyclades creates a Pool of IPs for each Network, and assigns a |
1194 |
unique IP address to each VM, thus connecting it to this Network. You can see |
1195 |
the IP pool of each network by running `snf-manage subnet-inspect |
1196 |
<subnet_ID>`. IP pools are automatically created and managed by Cyclades, |
1197 |
depending on the subnet of the Network. |
1198 |
* Bridges corresponding to physical VLANs, which are required for networks of |
1199 |
type `PRIVATE_PHYSICAL_VLAN`. |
1200 |
* One Bridge corresponding to one physical VLAN which is required for networks of |
1201 |
type `PRIVATE_MAC_PREFIX`. |
1202 |
|
1203 |
IPv4 addresses |
1204 |
************** |
1205 |
|
1206 |
An allocation pool of IPv4 addresses is automatically created for every network |
1207 |
with an IPv4 subnet. By default, the allocation pool contains the range of IP |
1208 |
addresses that are included in the subnet, except from the gateway and the |
1209 |
broadcast address of the network. The range of IP addresses can be restricted |
1210 |
using the `--allocation-pool` option of `snf-manage network-create` command. |
1211 |
The admin can externally reserve IP addresses to exclude them from automatic |
1212 |
allocation with the `--add-reserved-ips` option of `snf-manage network-modify` |
1213 |
command. For example the following command will reserve two IP addresses from |
1214 |
network with ID `42`: |
1215 |
|
1216 |
.. code-block:: console |
1217 |
|
1218 |
snf-manage network-modify --add-reserved-ips=10.0.0.21,10.0.0.22 42 |
1219 |
|
1220 |
.. warning:: Externally reserving IP addresses is also available at the Ganeti. |
1221 |
However, when using Cyclades with multiple Ganeti backends, the handling of |
1222 |
IP pools must be performed from Cyclades! |
1223 |
|
1224 |
Bridges |
1225 |
******* |
1226 |
|
1227 |
As already mentioned Cyclades use a pool of Bridges that must correspond |
1228 |
to Physical VLAN at the Ganeti level. A bridge from the pool is assigned to |
1229 |
each network of flavor `PHYSICAL_VLAN`. Creation of this pool is done |
1230 |
using `snf-manage pool-create` command. For example the following command |
1231 |
will create a pool containing the brdiges from `prv1` to `prv21`. |
1232 |
|
1233 |
.. code-block:: console |
1234 |
|
1235 |
# snf-manage pool-create --type=bridge --base=prv --size=20 |
1236 |
|
1237 |
You can verify the creation of the pool, and check its contents by running: |
1238 |
|
1239 |
.. code-block:: console |
1240 |
|
1241 |
# snf-manage pool-list |
1242 |
# snf-manage pool-show --type=bridge 1 |
1243 |
|
1244 |
Finally you can use the `pool-modify` management command in order to externally |
1245 |
reserve the values from pool, extend or shrink the pool if possible. |
1246 |
|
1247 |
MAC Prefixes |
1248 |
************ |
1249 |
|
1250 |
Cyclades also use a pool of MAC prefixes to assign to networks of flavor |
1251 |
`MAC_FILTERED`. Handling of this pool is done exactly as with pool of bridges, |
1252 |
except that the type option must be set to mac-prefix: |
1253 |
|
1254 |
.. code-block:: console |
1255 |
|
1256 |
# snf-manage pool-create --type=mac-prefix --base=aa:00:0 --size=65536 |
1257 |
|
1258 |
The above command will create a pool of MAC prefixes from ``aa:00:1`` to |
1259 |
``b9:ff:f``. The MAC prefix pool is responsible for providing only unicast and |
1260 |
locally administered MAC addresses, so many of these prefixes will be |
1261 |
externally reserved, to exclude from allocation. |
1262 |
|
1263 |
Quotas |
1264 |
~~~~~~ |
1265 |
|
1266 |
Handling of quotas for Cyclades resources is powered by Astakos quota |
1267 |
mechanism. During registration of Cyclades service to Astakos, the Cyclades |
1268 |
resources are also imported to Astakos for accounting and presentation. |
1269 |
|
1270 |
Upon a request that will result in a resource creation or removal, Cyclades |
1271 |
will communicate with Astakos to ensure that user quotas are within limits and |
1272 |
update the corresponding usage. If a limit is reached, the request will be |
1273 |
denied with an `overLimit(413)` fault. |
1274 |
|
1275 |
The resources that are exported by Cyclades are the following: |
1276 |
|
1277 |
* `cyclades.vm`: Number of virtual machines |
1278 |
* `cyclades.total_cpu`: Number of virtual machine processors |
1279 |
* `cyclades.cpu`: Number of virtual machine processors of running VMs |
1280 |
* `cyclades.total_ram`: Virtual machine memory size |
1281 |
* `cyclades.ram`: Virtual machine memory size of running VMs |
1282 |
* `cyclades.disk`: Virtual machine disk size |
1283 |
* `cyclades.floating_ip`: Number of floating IP addresses |
1284 |
* `cyclades.network.private`: Number of private virtual networks |
1285 |
|
1286 |
Enforcing quotas |
1287 |
~~~~~~~~~~~~~~~~ |
1288 |
|
1289 |
User quota can get overlimit, for example when a user is removed from a |
1290 |
project granting Cyclades resources. However, no action is automatically |
1291 |
taken to restrict users to their new limits. There is a special tool for |
1292 |
quota enforcement: |
1293 |
|
1294 |
.. code-block:: console |
1295 |
|
1296 |
# snf-manage enforce-resources-cyclades |
1297 |
|
1298 |
This command will check and report which users are overlimit on their |
1299 |
Cyclades quota; it will also suggest actions to be taken in order to enforce |
1300 |
quota limits, dependent on the overlimit resource: |
1301 |
|
1302 |
* `cyclades.vm`: Delete VMs |
1303 |
* `cyclades.total_cpu`: Delete VMs |
1304 |
* `cyclades.cpu`: Shutdown VMs |
1305 |
* `cyclades.total_ram`: Delete VMs |
1306 |
* `cyclades.ram`: Shutdown VMs |
1307 |
* `cyclades.disk`: Delete VMs |
1308 |
* `cyclades.floating_ip`: Detach and remove IPs |
1309 |
|
1310 |
VMs to be deleted/shutdown are chosen first by state in the following order: |
1311 |
ERROR, BUILD, STOPPED, STARTED or RESIZE and then by decreasing ID. When |
1312 |
needing to remove IPs, we first choose IPs that are free, then those |
1313 |
attached to VMs, using the same VM ordering. |
1314 |
|
1315 |
By default, the command checks only the following resources: `cyclades.cpu`, |
1316 |
`cyclades.ram`, and `cyclades.floating_ip`; that is, the less dangerous |
1317 |
ones, those that do not result in *deleting* any VM. One can change the |
1318 |
default behavior by specifying the desired resources with option |
1319 |
``--resources``. It is also possible to specify users to be checked or |
1320 |
excluded. |
1321 |
|
1322 |
Actual enforcement is done with option ``--fix``. In order to control the |
1323 |
load that quota enforcement may cause on Cyclades, one can limit the number |
1324 |
of operations per backend. For example, |
1325 |
|
1326 |
.. code-block:: console |
1327 |
|
1328 |
# snf-manage enforce-resources-cyclades --fix --max-operations 10 |
1329 |
|
1330 |
will apply only the first 10 listed actions per backend. One can repeat the |
1331 |
operation, until nothing is left to be done. |
1332 |
|
1333 |
To control load a timeout can also be set for shutting down VMs (using |
1334 |
option ``--shutdown-timeout <sec>``). This may be needed to avoid |
1335 |
expensive operations triggered by shutdown, such as Windows updates. |
1336 |
|
1337 |
The command outputs the list of applied actions and reports whether each |
1338 |
action succeeded or not. Failure is reported if for any reason cyclades |
1339 |
failed to process the job and submit it to the backend. |
1340 |
|
1341 |
Cyclades advanced operations |
1342 |
---------------------------- |
1343 |
|
1344 |
Reconciliation mechanism |
1345 |
~~~~~~~~~~~~~~~~~~~~~~~~ |
1346 |
|
1347 |
Cyclades - Ganeti reconciliation |
1348 |
```````````````````````````````` |
1349 |
|
1350 |
On certain occasions, such as a Ganeti or RabbitMQ failure, the state of |
1351 |
Cyclades database may differ from the real state of VMs and networks in the |
1352 |
Ganeti backends. The reconciliation process is designed to synchronize the |
1353 |
state of the Cyclades DB with Ganeti. There are two management commands for |
1354 |
reconciling VMs and Networks that will detect stale, orphans and out-of-sync |
1355 |
VMs and networks. To fix detected inconsistencies, use the `--fix-all`. |
1356 |
|
1357 |
.. code-block:: console |
1358 |
|
1359 |
$ snf-manage reconcile-servers |
1360 |
$ snf-manage reconcile-servers --fix-all |
1361 |
|
1362 |
$ snf-manage reconcile-networks |
1363 |
$ snf-manage reconcile-networks --fix-all |
1364 |
|
1365 |
Please see ``snf-manage reconcile-servers --help`` and ``snf-manage |
1366 |
reconcile--networks --help`` for all the details. |
1367 |
|
1368 |
|
1369 |
Cyclades - Astakos reconciliation |
1370 |
````````````````````````````````` |
1371 |
|
1372 |
As already mentioned, Cyclades communicates with Astakos for resource |
1373 |
accounting and quota enforcement. In rare cases, e.g. unexpected |
1374 |
failures, the two services may get unsynchronized. For this reason there |
1375 |
are the `reconcile-commissions-cyclades` and `reconcile-resources-cyclades` |
1376 |
command that will synchronize the state of the two services. The first |
1377 |
command will detect any pending commissions, while the second command will |
1378 |
detect that the usage that is reported by Astakos is correct. |
1379 |
To fix detected inconsistencies, use the `--fix` option. |
1380 |
|
1381 |
.. code-block:: console |
1382 |
|
1383 |
$ snf-manage reconcile-commissions-cyclades |
1384 |
$ snf-manage reconcile-commissions-cyclades --fix |
1385 |
|
1386 |
$ snf-manage reconcile-resources-cyclades |
1387 |
$ snf-manage reconcile-resources-cyclades --fix |
1388 |
|
1389 |
|
1390 |
Cyclades resources reconciliation |
1391 |
````````````````````````````````` |
1392 |
|
1393 |
Reconciliation of pools will check the consistency of available pools by |
1394 |
checking that the values from each pool are not used more than once, and also |
1395 |
that the only reserved values in a pool are the ones used. Pool reconciliation |
1396 |
will check pools of bridges, MAC prefixes, and IPv4 addresses for all networks. |
1397 |
To fix detected inconsistencies, use the `--fix` option. |
1398 |
|
1399 |
|
1400 |
.. code-block:: console |
1401 |
|
1402 |
$ snf-manage reconcile-pools |
1403 |
$ snf-manage reconcile-pools --fix |
1404 |
|
1405 |
.. _admin-guide-stats: |
1406 |
|
1407 |
VM stats collecting |
1408 |
~~~~~~~~~~~~~~~~~~~ |
1409 |
|
1410 |
snf-cyclades-gtools comes with a collectd plugin to collect CPU and network |
1411 |
stats for Ganeti VMs and an example collectd configuration. snf-stats-app is a |
1412 |
Django (snf-webproject) app that serves the VM stats graphs by reading the VM |
1413 |
stats (from RRD files) and serves graphs. |
1414 |
|
1415 |
The snf-stats-app was originally written by `GRNET NOC <http://noc.grnet.gr>`_ |
1416 |
as a WSGI Python app and was ported to a Synnefo (snf-webproject) app. |
1417 |
|
1418 |
snf-stats-app configuration |
1419 |
``````````````````````````` |
1420 |
|
1421 |
The snf-stats-app node should have collectd installed. The collectd |
1422 |
configuration should enable the network plugin, assuming the server role, and |
1423 |
the RRD plugin / backend, to store the incoming stats. Your |
1424 |
``/etc/collectd/collectd.conf`` should look like: |
1425 |
|
1426 |
.. code-block:: console |
1427 |
|
1428 |
FQDNLookup true |
1429 |
LoadPlugin syslog |
1430 |
<Plugin syslog> |
1431 |
LogLevel info |
1432 |
</Plugin> |
1433 |
|
1434 |
LoadPlugin network |
1435 |
LoadPlugin rrdtool |
1436 |
<Plugin network> |
1437 |
TimeToLive 128 |
1438 |
<Listen "okeanos.io" "25826"> |
1439 |
SecurityLevel "Sign" |
1440 |
AuthFile "/etc/collectd/passwd" |
1441 |
</Listen> |
1442 |
|
1443 |
ReportStats false |
1444 |
MaxPacketSize 65535 |
1445 |
</Plugin> |
1446 |
|
1447 |
|
1448 |
<Plugin rrdtool> |
1449 |
DataDir "/var/lib/collectd/rrd" |
1450 |
CacheTimeout 120 |
1451 |
CacheFlush 900 |
1452 |
WritesPerSecond 30 |
1453 |
RandomTimeout 0 |
1454 |
</Plugin> |
1455 |
|
1456 |
Include "/etc/collectd/filters.conf" |
1457 |
Include "/etc/collectd/thresholds.conf" |
1458 |
|
1459 |
|
1460 |
An example collectd config file is provided in |
1461 |
``/usr/share/doc/snf-stats-app/examples/stats-colletcd.conf``. |
1462 |
|
1463 |
The recommended deployment is to run snf-stats-app using gunicorn with an |
1464 |
Apache2 or nginx reverse proxy (using the same configuration as the other |
1465 |
Synnefo services / apps). An example gunicorn config file is provided in |
1466 |
``/usr/share/doc/snf-stats-app/examples/stats.gunicorn``. |
1467 |
|
1468 |
Make sure to edit the settings under |
1469 |
``/etc/synnefo/20-snf-stats-app-settings.conf`` to match your deployment. |
1470 |
More specifically, you should change the ``STATS_BASE_URL`` setting (refer |
1471 |
to previous documentation on the BASE_URL settings used by the other Synnefo |
1472 |
services / apps) and the ``RRD_PREFIX`` and ``GRAPH_PREFIX`` settings. |
1473 |
|
1474 |
You should also set the ``STATS_SECRET_KEY`` to a random string and make sure |
1475 |
it's the same at the ``CYCLADES_STATS_SECRET_KEY`` on the Cyclades host (see |
1476 |
below). |
1477 |
|
1478 |
``RRD_PREFIX`` is the directory where collectd stores the RRD files. The |
1479 |
default setting matches the default RRD directory for the collectd RRDtool |
1480 |
plugin. In a more complex setup, the collectd daemon could run on a separate |
1481 |
host and export the RRD directory to the snf-stats-app node via e.g. NFS. |
1482 |
|
1483 |
``GRAPH_PREFIX`` is the directory where collectd stores the resulting |
1484 |
stats graphs. You should create it manually, in case it doesn't exist. |
1485 |
|
1486 |
.. code-block:: |
1487 |
|
1488 |
# mkdir /var/cache/snf-stats-app/ |
1489 |
# chown www-data:wwwdata /var/cache/snf-stats-app/ |
1490 |
|
1491 |
The snf-stats-app will typically run as the ``www-data`` user. In that case, |
1492 |
make sure that the ``www-data`` user should have read access to the |
1493 |
``RRD_PREFIX`` directory and read / write access to the ``GRAPH_PREFIX`` |
1494 |
directory. |
1495 |
|
1496 |
snf-stats-app, based on the ``STATS_BASE_URL`` setting will export the |
1497 |
following URL 'endpoints`: |
1498 |
* CPU stats bar: ``STATS_BASE_URL``/v1.0/cpu-bar/<encrypted VM hostname> |
1499 |
* Network stats bar: ``STATS_BASE_URL``/v1.0/net-bar/<encrypted VM hostname> |
1500 |
* CPU stats daily graph: ``STATS_BASE_URL``/v1.0/cpu-ts/<encrypted VM hostname> |
1501 |
* Network stats daily graph: ``STATS_BASE_URL``/v1.0/net-ts/<encrypted VM hostname> |
1502 |
* CPU stats weekly graph: ``STATS_BASE_URL``/v1.0/cpu-ts-w/<encrypted VM hostname> |
1503 |
* Network stats weekly graph: ``STATS_BASE_URL``/v1.0/net-ts-w/<encrypted VM hostname> |
1504 |
|
1505 |
You can verify that these endpoints are exported by issuing: |
1506 |
|
1507 |
.. code-block:: |
1508 |
|
1509 |
# snf-manage show_urls |
1510 |
|
1511 |
snf-cyclades-gtools configuration |
1512 |
````````````````````````````````` |
1513 |
|
1514 |
To enable VM stats collecting, you will need to: |
1515 |
* Install collectd on the every Ganeti (VM-capable) node. |
1516 |
* Enable the Ganeti stats plugin in your collectd configuration. This can be |
1517 |
achived by either copying the example collectd conf file that comes with |
1518 |
snf-cyclades-gtools |
1519 |
(``/usr/share/doc/snf-cyclades-gtools/examples/ganeti-stats-collectd.conf``) |
1520 |
or by adding the following line to your existing (or default) collectd |
1521 |
conf file: |
1522 |
|
1523 |
Include /etc/collectd/ganeti-stats.conf |
1524 |
|
1525 |
In the latter case, make sure to configure collectd to send the collected |
1526 |
stats to your collectd server (via the network plugin). For more details on |
1527 |
how to do this, check the collectd example config file provided by the |
1528 |
package and the collectd documentation. |
1529 |
|
1530 |
snf-cyclades-app configuration |
1531 |
`````````````````````````````` |
1532 |
|
1533 |
At this point, stats collecting should be enabled and working. You can check |
1534 |
that everything is ok by checking the contents of ``/var/lib/collectd/rrd/`` |
1535 |
directory (it will gradually get populated with directories containing RRD |
1536 |
files / stats for every Synnefo instances). |
1537 |
|
1538 |
You should also check that gunicorn and Apache2 are configured correctly by |
1539 |
accessing the graph URLs for a VM (whose stats have been populated in |
1540 |
``/var/lib/collectd/rrd``). |
1541 |
|
1542 |
Cyclades uses the ``CYCLADES_STATS_SECRET_KEY`` setting in |
1543 |
``20-snf-cyclades-app`` to encrypt the instance hostname in the stats graph |
1544 |
URL. This settings should be set to a random value and match the |
1545 |
``STATS_SECRET_KEY`` on the Stats host. |
1546 |
|
1547 |
Cyclades (snf-cyclades-app) fetches the stat graphs for VMs based on four |
1548 |
settings in ``20-snf-cyclades-app-api.conf``. The settings are: |
1549 |
|
1550 |
* CPU_BAR_GRAPH_URL = 'https://stats.host/stats/v1.0/cpu-bar/%s' |
1551 |
* CPU_TIMESERIES_GRAPH_URL = 'https://stats.host/stats/v1.0/cpu-ts/%s' |
1552 |
* NET_BAR_GRAPH_URL = 'https://stats.host/stats/v1.0/net-bar/%s' |
1553 |
* NET_TIMESERIES_GRAPH_URL = 'https://stats.host/stats/v1.0/net-ts/%s' |
1554 |
|
1555 |
Make sure that you change this settings to match your ``STATS_BASE_URL`` |
1556 |
(and generally the Apache2 / gunicorn deployment on your stats host). |
1557 |
|
1558 |
Cyclades will pass these URLs to the Cyclades UI and the user's browser will |
1559 |
fetch them when needed. |
1560 |
|
1561 |
|
1562 |
Helpdesk |
1563 |
-------- |
1564 |
|
1565 |
Helpdesk application provides the ability to view the virtual servers and |
1566 |
networks of all users, along with the ability to perform some basic actions |
1567 |
like administratively suspending a server. You can perform look-ups by |
1568 |
user UUID or email, by server ID (vm-$id) or by an IPv4 address. |
1569 |
|
1570 |
If you want to activate the helpdesk application you can set to `True` the |
1571 |
`HELPDESK_ENABLED` setting. Access to helpdesk views (under |
1572 |
`$BASE_URL/helpdesk`) is only to allowed to users that belong to Astakos |
1573 |
groups defined in the `HELPDESK_PERMITTED_GROUPS` setting, which by default |
1574 |
contains the `helpdesk` group. For example, to allow <user_id> |
1575 |
to access helpdesk view, you should run the following command in the Astakos |
1576 |
node: |
1577 |
|
1578 |
.. code-block:: console |
1579 |
|
1580 |
snf-manage group-add helpdesk |
1581 |
snf-manage user-modify --add-group=helpdesk <user_id> |
1582 |
|
1583 |
|
1584 |
Cyclades internals |
1585 |
------------------ |
1586 |
|
1587 |
Asynchronous communication with Ganeti backends |
1588 |
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
1589 |
Synnefo uses Google Ganeti backends for VM cluster management. In order for |
1590 |
Cyclades to be able to handle thousands of user requests, Cyclades and Ganeti |
1591 |
communicate asynchronously. Briefly, requests are submitted to Ganeti through |
1592 |
Ganeti's RAPI/HTTP interface, and then asynchronous notifications about the |
1593 |
progress of Ganeti jobs are being created and pushed upwards to Cyclades. The |
1594 |
architecture and communication with a Ganeti backend is shown in the graph |
1595 |
below: |
1596 |
|
1597 |
.. image:: images/cyclades-ganeti-communication.png |
1598 |
:width: 40% |
1599 |
:target: _images/cyclades-ganeti-communication.png |
1600 |
|
1601 |
The Cyclades API server is responsible for handling user requests. Read-only |
1602 |
requests are directly served by looking up the Cyclades DB. If the request |
1603 |
needs an action in the Ganeti backend, Cyclades submit jobs to the Ganeti |
1604 |
master using the `Ganeti RAPI interface |
1605 |
<http://docs.ganeti.org/ganeti/2.8/html/rapi.html>`_. |
1606 |
|
1607 |
While Ganeti executes the job, `snf-ganeti-eventd`, and `snf-progress-monitor` |
1608 |
are monitoring the progress of the job and send corresponding messages to the |
1609 |
RabbitMQ servers. These components are part of `snf-cyclades-gtools` and must |
1610 |
be installed on all Ganeti nodes. Specially: |
1611 |
|
1612 |
* *snf-ganeti-eventd* sends messages about operations affecting the operating |
1613 |
state of instances and networks. Works by monitoring the Ganeti job queue. |
1614 |
* *snf-progress_monitor* sends messages about the progress of the Image deployment |
1615 |
phase which is done by the Ganeti OS Definition `snf-image`. |
1616 |
|
1617 |
Finally, `snf-dispatcher` consumes messages from the RabbitMQ queues, processes |
1618 |
these messages and properly updates the state of the Cyclades DB. Subsequent |
1619 |
requests to the Cyclades API, will retrieve the updated state from the DB. |
1620 |
|
1621 |
|
1622 |
List of all Synnefo components |
1623 |
============================== |
1624 |
|
1625 |
They are also available from our apt repository: ``apt.dev.grnet.gr`` |
1626 |
|
1627 |
* `snf-common <http://www.synnefo.org/docs/snf-common/latest/index.html>`_ |
1628 |
* `snf-webproject <http://www.synnefo.org/docs/snf-webproject/latest/index.html>`_ |
1629 |
* `snf-astakos-app <http://www.synnefo.org/docs/astakos/latest/index.html>`_ |
1630 |
* `snf-pithos-backend <http://www.synnefo.org/docs/pithos/latest/backends.html>`_ |
1631 |
* `snf-pithos-app <http://www.synnefo.org/docs/pithos/latest/index.html>`_ |
1632 |
* `snf-pithos-webclient <http://www.synnefo.org/docs/pithos-webclient/latest/index.html>`_ |
1633 |
* `snf-cyclades-app <http://www.synnefo.org/docs/snf-cyclades-app/latest/index.html>`_ |
1634 |
* `snf-cyclades-gtools <http://www.synnefo.org/docs/snf-cyclades-gtools/latest/index.html>`_ |
1635 |
* `astakosclient <http://www.synnefo.org/docs/astakosclient/latest/index.html>`_ |
1636 |
* `snf-vncauthproxy <https://code.grnet.gr/projects/vncauthproxy>`_ |
1637 |
* `snf-image <http://www.synnefo.org/docs/snf-image/latest/index.html/>`_ |
1638 |
* `snf-image-creator <http://www.synnefo.org/docs/snf-image-creator/latest/index.html>`_ |
1639 |
* `snf-occi <http://www.synnefo.org/docs/snf-occi/latest/index.html>`_ |
1640 |
* `snf-cloudcms <http://www.synnefo.org/docs/snf-cloudcms/latest/index.html>`_ |
1641 |
* `nfdhcpd <https://code.grnet.gr/projects/nfdhcpd>`_ |
1642 |
|
1643 |
|
1644 |
Synnefo management commands ("snf-manage") |
1645 |
========================================== |
1646 |
|
1647 |
Each Synnefo service, Astakos, Pithos and Cyclades are controlled by the |
1648 |
administrator using the "snf-manage" admin tool. This tool is an extension of |
1649 |
the Django command-line management utility. It is run on the host that runs |
1650 |
each service and provides different types of commands depending the services |
1651 |
running on the host. If you are running more than one service on the same host |
1652 |
"snf-manage" adds all the corresponding commands for each service dynamically, |
1653 |
providing a unified admin environment. |
1654 |
|
1655 |
To run "snf-manage" you just type: |
1656 |
|
1657 |
.. code-block:: console |
1658 |
|
1659 |
# snf-manage <command> [arguments] |
1660 |
|
1661 |
on the corresponding host that runs the service. For example, if you have all |
1662 |
services running on different physical hosts you would do: |
1663 |
|
1664 |
.. code-block:: console |
1665 |
|
1666 |
root@astakos-host # snf-manage <astakos-command> [argument] |
1667 |
root@pithos-host # snf-manage <pithos-command> [argument] |
1668 |
root@cyclades-host # snf-manage <cyclades-command> [argument] |
1669 |
|
1670 |
If you have all services running on the same host you would do: |
1671 |
|
1672 |
.. code-block:: console |
1673 |
|
1674 |
root@synnefo-host # snf-manage <{astakos,pithos,cyclades}-command> [argument] |
1675 |
|
1676 |
Note that you cannot execute a service's command on a host that is not running |
1677 |
this service. For example, the following will return an error if Astakos and |
1678 |
Cyclades are installed on different physical hosts: |
1679 |
|
1680 |
.. code-block:: console |
1681 |
|
1682 |
root@astakos-host # snf-manage <cyclades-command> [argument] |
1683 |
Unknown command: 'cyclades-command' |
1684 |
Type 'snf-manage help' for usage. |
1685 |
|
1686 |
This is the complete list of "snf-manage" commands for each service. |
1687 |
|
1688 |
Astakos snf-manage commands |
1689 |
--------------------------- |
1690 |
|
1691 |
============================ =========================== |
1692 |
Name Description |
1693 |
============================ =========================== |
1694 |
fix-superusers Transform superusers created by syncdb into AstakosUser instances |
1695 |
cleanup-full Cleanup sessions and session catalog |
1696 |
commission-list List pending commissions |
1697 |
commission-show Show details for a pending commission |
1698 |
component-add Register a component |
1699 |
component-list List components |
1700 |
component-modify Modify component attributes |
1701 |
component-show Show component details |
1702 |
project-control Manage projects and applications |
1703 |
project-list List projects |
1704 |
project-show Show project details |
1705 |
quota-list List user quota |
1706 |
quota-verify Check the integrity of user quota |
1707 |
reconcile-resources-astakos Reconcile resource usage of Quotaholder with Astakos DB |
1708 |
resource-list List resources |
1709 |
resource-modify Modify a resource's default base quota and boolean flags |
1710 |
service-export-astakos Export Astakos services and resources in JSON format |
1711 |
service-import Register services |
1712 |
service-list List services |
1713 |
service-show Show service details |
1714 |
term-add Add approval terms |
1715 |
user-activation-send Send user activation |
1716 |
user-add Add user |
1717 |
authpolicy-add Create a new authentication provider policy profile |
1718 |
authpolicy-list List existing authentication provider policy profiles |
1719 |
authpolicy-remove Remove an authentication provider policy |
1720 |
authpolicy-set Assign an existing authentication provider policy profile to a user or group |
1721 |
authpolicy-show Show authentication provider profile details |
1722 |
group-add Create a group with the given name |
1723 |
group-list List available groups |
1724 |
user-list List users |
1725 |
user-modify Modify user |
1726 |
user-show Show user details |
1727 |
oauth2-client-add Create an oauth2 client |
1728 |
oauth2-client-list List oauth2 clients |
1729 |
oauth2-client-remove Remove an oauth2 client along with its registered redirect urls |
1730 |
============================ =========================== |
1731 |
|
1732 |
Pithos snf-manage commands |
1733 |
-------------------------- |
1734 |
|
1735 |
============================ =========================== |
1736 |
Name Description |
1737 |
============================ =========================== |
1738 |
reconcile-commissions-pithos Display unresolved commissions and trigger their recovery |
1739 |
service-export-pithos Export Pithos services and resources in JSON format |
1740 |
reconcile-resources-pithos Detect unsynchronized usage between Astakos and Pithos DB resources and synchronize them if specified so. |
1741 |
file-show Display object information |
1742 |
============================ =========================== |
1743 |
|
1744 |
Cyclades snf-manage commands |
1745 |
---------------------------- |
1746 |
|
1747 |
============================== =========================== |
1748 |
Name Description |
1749 |
============================== =========================== |
1750 |
backend-add Add a new Ganeti backend |
1751 |
backend-list List backends |
1752 |
backend-modify Modify a backend |
1753 |
backend-update-status Update backend statistics for instance allocation |
1754 |
backend-remove Remove a Ganeti backend |
1755 |
enforce-resources-cyclades Check and fix quota violations for Cyclades resources |
1756 |
server-create Create a new server |
1757 |
server-show Show server details |
1758 |
server-list List servers |
1759 |
server-modify Modify a server |
1760 |
server-import Import an existing Ganeti VM into synnefo |
1761 |
server-inspect Inspect a server in DB and Ganeti |
1762 |
network-create Create a new network |
1763 |
network-list List networks |
1764 |
network-modify Modify a network |
1765 |
network-inspect Inspect network state in DB and Ganeti |
1766 |
network-remove Delete a network |
1767 |
flavor-create Create a new flavor |
1768 |
flavor-list List flavors |
1769 |
flavor-modify Modify a flavor |
1770 |
image-list List images |
1771 |
image-show Show image details |
1772 |
pool-create Create a bridge or mac-prefix pool |
1773 |
pool-show Show pool details |
1774 |
pool-list List pools |
1775 |
pool-modify Modify a pool |
1776 |
pool-remove Delete a pool |
1777 |
port-create Create a port connecting a server to a network |
1778 |
port-inspect Inspect the state of a port in DB and Ganeti |
1779 |
port-list List ports |
1780 |
port-remove Delete a port |
1781 |
floating-ip-create Create a new floating IP |
1782 |
floating-ip-attach Attach a floating IP to a server |
1783 |
floating-ip-detach Detach a floating IP from a server |
1784 |
floating-ip-list List floating IPs |
1785 |
floating-ip-remove Delete a floating IP |
1786 |
queue-inspect Inspect the messages of a RabbitMQ queue |
1787 |
queue-retry Resend messages from Dead Letter queues to original exchanges |
1788 |
service-export-cyclades Export Cyclades services and resources in JSON format |
1789 |
subnet-create Create a subnet |
1790 |
subnet-inspect Inspect a subnet in DB |
1791 |
subnet-list List subnets |
1792 |
subnet-modify Modify a subnet |
1793 |
reconcile-servers Reconcile servers of Synnefo DB with state of Ganeti backend |
1794 |
reconcile-networks Reconcile networks of Synnefo DB with state of Ganeti backend |
1795 |
reconcile-pools Check consistency of pool resources |
1796 |
reconcile-commissions-cyclades Detect and resolve pending commissions to Quotaholder |
1797 |
reconcile-resources-cyclades Reconcile resource usage of Astakos with Cyclades DB. |
1798 |
============================== =========================== |
1799 |
|
1800 |
|
1801 |
Astakos helper scripts |
1802 |
====================== |
1803 |
|
1804 |
Astakos includes two scripts to facilitate the installation procedure. |
1805 |
Running: |
1806 |
|
1807 |
.. code-block:: console |
1808 |
|
1809 |
snf-component-register [<component_name>] |
1810 |
|
1811 |
automates the registration of the standard Synnefo components (astakos, |
1812 |
cyclades, and pithos) in astakos database. It internally uses the script: |
1813 |
|
1814 |
.. code-block:: console |
1815 |
|
1816 |
snf-service-export <component_name> <base_url> |
1817 |
|
1818 |
which simulates the export of service and resource definitions of the |
1819 |
standard Synnefo components. |
1820 |
|
1821 |
|
1822 |
Pithos managing accounts |
1823 |
======================== |
1824 |
|
1825 |
Pithos provides a utility tool for managing accounts. |
1826 |
To run you just type: |
1827 |
|
1828 |
.. code-block:: console |
1829 |
|
1830 |
# pithos-manage-accounts <command> [arguments] |
1831 |
|
1832 |
This is the list of the available commands: |
1833 |
|
1834 |
============================ =========================== |
1835 |
Name Description |
1836 |
============================ =========================== |
1837 |
delete Remove an account from the Pithos DB |
1838 |
export-quota Export account quota in a file |
1839 |
list List existing/dublicate accounts |
1840 |
merge Move an account contents in another account |
1841 |
set-container-quota Set container quota for all or a specific account |
1842 |
============================ =========================== |
1843 |
|
1844 |
|
1845 |
The "kamaki" API client |
1846 |
======================= |
1847 |
|
1848 |
To upload, register or modify an image you will need the **kamaki** tool. |
1849 |
Before proceeding make sure that it is configured properly. Verify that |
1850 |
*image.url*, *file.url*, *user.url* and *token* are set as needed: |
1851 |
|
1852 |
.. code-block:: console |
1853 |
|
1854 |
$ kamaki config list |
1855 |
|
1856 |
To change a setting use ``kamaki config set``: |
1857 |
|
1858 |
.. code-block:: console |
1859 |
|
1860 |
$ kamaki config set cloud.default.url https://example.com/identity/v2.0 |
1861 |
$ kamaki config set cloud.default.token ... |
1862 |
|
1863 |
To test that everything works, try authenticating the current account with |
1864 |
kamaki: |
1865 |
|
1866 |
.. code-block:: console |
1867 |
|
1868 |
$ kamaki user authenticate |
1869 |
|
1870 |
This will output user information. |
1871 |
|
1872 |
Upload Image |
1873 |
------------ |
1874 |
|
1875 |
By convention, images are stored in a container called ``images``. Check if the |
1876 |
container exists, by listing all containers in your account: |
1877 |
|
1878 |
.. code-block:: console |
1879 |
|
1880 |
$ kamaki file list /images |
1881 |
|
1882 |
If the container ``images`` does not exist, create it: |
1883 |
|
1884 |
.. code-block:: console |
1885 |
|
1886 |
$ kamaki container create images |
1887 |
|
1888 |
You are now ready to upload an image to container ``images``. You can upload it |
1889 |
with a Pithos client, or use kamaki directly: |
1890 |
|
1891 |
.. code-block:: console |
1892 |
|
1893 |
$ kamaki file upload ubuntu.iso /images |
1894 |
|
1895 |
You can use any Pithos client to verify that the image was uploaded correctly, |
1896 |
or you can list the contents of the container with kamaki: |
1897 |
|
1898 |
.. code-block:: console |
1899 |
|
1900 |
$ kamaki file list /images |
1901 |
|
1902 |
The full Pithos URL for the previous example will be |
1903 |
``pithos://u53r-un1qu3-1d/images/ubuntu.iso`` where ``u53r-un1qu3-1d`` is the |
1904 |
unique user id (uuid). |
1905 |
|
1906 |
Register Image |
1907 |
-------------- |
1908 |
|
1909 |
To register an image you will need to use the full or the relative Pithos URL. |
1910 |
To register as a public image the one from the previous example use: |
1911 |
|
1912 |
.. code-block:: console |
1913 |
|
1914 |
$ kamaki image register --name=Ubuntu --location=/images/ubuntu.iso --public |
1915 |
|
1916 |
The ``--public`` flag is important, if missing the registered image will not |
1917 |
be listed by ``kamaki image list``. |
1918 |
|
1919 |
Use ``kamaki image register`` with no arguments to see a list of available |
1920 |
options. A more complete example would be the following: |
1921 |
|
1922 |
.. code-block:: console |
1923 |
|
1924 |
$ kamaki image register --name Ubuntu --location /images/ubuntu.iso \ |
1925 |
--public --disk-format diskdump --property kernel=3.1.2 |
1926 |
|
1927 |
To verify that the image was registered successfully use: |
1928 |
|
1929 |
.. code-block:: console |
1930 |
|
1931 |
$ kamaki image list --name-like ubuntu |
1932 |
|
1933 |
|
1934 |
Miscellaneous |
1935 |
============= |
1936 |
|
1937 |
.. _branding: |
1938 |
|
1939 |
Branding |
1940 |
-------- |
1941 |
|
1942 |
Since Synnefo v0.14, you are able to adapt the Astakos, Pithos and Cyclades Web |
1943 |
UI to your company’s visual identity. This is possible using the snf-branding |
1944 |
component, which is automatically installed on the nodes running the API |
1945 |
servers for Astakos, Pithos and Cyclades. |
1946 |
|
1947 |
Configuration |
1948 |
~~~~~~~~~~~~~ |
1949 |
|
1950 |
This can be done by modifing the settings provided by the snf-branding component |
1951 |
to match your service identity. The settings for the snf-branding application |
1952 |
can be found inside the configuration file ``/etc/synnefo/15-snf-branding.conf`` |
1953 |
on the nodes that have Astakos, Pithos and Cyclades installed. |
1954 |
|
1955 |
By default, the global service name is "Synnefo" and the company name is |
1956 |
"GRNET". These names and their respective logos and URLs are used throughout |
1957 |
the Astakos, Pithos and Cyclades UI. |
1958 |
|
1959 |
**Names and URLs:** |
1960 |
|
1961 |
The first group of branding customization refers to the service's and company's |
1962 |
information. |
1963 |
|
1964 |
You can overwrite the company and the service name and URL respectively by |
1965 |
uncommenting and setting the following: |
1966 |
|
1967 |
.. code-block:: python |
1968 |
|
1969 |
# setting used in Astakos Dashboard/Projects pages |
1970 |
BRANDING_SERVICE_NAME = 'My cloud' |
1971 |
BRANDING_SERVICE_URL = 'http://www.mycloud.synnefo.org/' |
1972 |
|
1973 |
# settings used in Astakos, Pithos, Cyclades footer only if |
1974 |
# BRANDING_SHOW_COPYRIGHT is set to True |
1975 |
BRANDING_SHOW_COPYRIGHT = True |
1976 |
BRANDING_COMPANY_NAME = 'Company LTD' |
1977 |
BRANDING_COMPANY_URL = 'https://www.company-ltd.synnefo.org/' |
1978 |
|
1979 |
|
1980 |
**Copyright and footer options:** |
1981 |
|
1982 |
By default, no Copyright message is shown in the UI footer. If you want to make |
1983 |
it visible in the footer of Astakos, Pithos and Cyclades UI, you can uncomment |
1984 |
and set to ``True`` the ``BRANDING_SHOW_COPYRIGHT`` setting: |
1985 |
|
1986 |
.. code-block:: python |
1987 |
|
1988 |
#BRANDING_SHOW_COPYRIGHT = False |
1989 |
|
1990 |
Copyright message defaults to 'Copyright (c) 2011-<current_year> |
1991 |
<BRANDING_COMPANY_NAME>.' but you can overwrite it to a completely custom one by |
1992 |
setting the following option: |
1993 |
|
1994 |
.. code-block:: python |
1995 |
|
1996 |
BRANDING_COPYRIGHT_MESSAGE = 'Copyright (c) 2011-2013 GRNET' |
1997 |
|
1998 |
If you want to include a custom message in the footer, you can uncomment and |
1999 |
set the ``BRANDING_FOOTER_EXTRA_MESSAGE`` setting. You can use html markup. |
2000 |
Your custom message will appear above Copyright message at the Compute |
2001 |
templates and the Dashboard UI. |
2002 |
|
2003 |
.. code-block:: python |
2004 |
|
2005 |
#BRANDING_FOOTER_EXTRA_MESSAGE = '' |
2006 |
|
2007 |
|
2008 |
**Images:** |
2009 |
|
2010 |
The Astakos, Pithos and Cyclades Web UI has some logos and images. |
2011 |
|
2012 |
The branding-related images are presented in the following table: |
2013 |
|
2014 |
=============== ============================ ========= |
2015 |
Image Name/extension convention Usage |
2016 |
=============== ============================ ========= |
2017 |
Favicon favicon.ico Favicon for all services |
2018 |
Dashboard logo dashboard_logo.png Visible in all Astakos UI pages |
2019 |
Compute logo compute_logo.png Visible in all Cyclades UI pages |
2020 |
Console logo console_logo.png Visible in the Cyclades Console Window |
2021 |
Storage logo storage_logo.png Visible in all Pithos UI pages |
2022 |
=============== ============================ ========= |
2023 |
|
2024 |
There are two methods available for replacing all, or individual, |
2025 |
branding-related images: |
2026 |
|
2027 |
1. Create a new directory inside ``/usr/share/synnefo/static/`` (e.g. |
2028 |
``mybranding``) and place there some or all of your images. |
2029 |
|
2030 |
If you want to replace all of your images, keep the name/extension |
2031 |
conventions as indicated in the above table and change the |
2032 |
``BRANDING_IMAGE_MEDIA_URL`` setting accordingly: |
2033 |
|
2034 |
.. code-block:: python |
2035 |
|
2036 |
# using relative path |
2037 |
BRANDING_IMAGE_MEDIA_URL= '/static/mybranding/images/' |
2038 |
|
2039 |
# or if you already host them in a separate domain (e.g. cdn) |
2040 |
BRANDING_IMAGE_MEDIA_URL= 'https://cdn.synnefo.org/branding/images/' |
2041 |
|
2042 |
|
2043 |
If you wish to replace individual images, **do not uncomment** |
2044 |
``BRANDING_IMAGE_MEDIA_URL``, but instead provide a relative path, pointing to |
2045 |
the file inside your directory for each ``BRANDING_<image>_URL`` that you wish |
2046 |
to replace. |
2047 |
|
2048 |
2. Upload some or all of your images to a server and replace each |
2049 |
``BRANDING_<image>_URL`` with the absolute url of the image (i.e. |
2050 |
``BRANDING_DASHBOARD_URL = 'https://www.synnefo.com/images/my_dashboard.jpg'``). |
2051 |
|
2052 |
Note that the alternative text for each image tag inside html documents is |
2053 |
alt=“BRANDING_SERVICE_NAME {Dashboard, Compute. Console, Storage}” respectively. |
2054 |
|
2055 |
.. note:: Retina optimized images: |
2056 |
|
2057 |
Synnefo UI is optimized for Retina displays. As far as images are concerned, |
2058 |
`retina.js <http://retinajs.com/>`_ is used. |
2059 |
|
2060 |
Retina.js checks each image on a page to see if there is a high-resolution |
2061 |
version of that image on your server. If a high-resolution variant exists, |
2062 |
the script will swap in that image in-place. |
2063 |
|
2064 |
The script assumes you use `Apple's prescribed high-resolution modifier (@2x) |
2065 |
<http://developer.apple.com/library/ios/#documentation/2DDrawing/Conceptual/ |
2066 |
DrawingPrintingiOS/SupportingHiResScreensInViews/SupportingHiResScreensInViews |
2067 |
.html#//apple_ref/doc/uid/TP40010156-CH15-SW1>`_ to denote high-resolution |
2068 |
image variants on your server. |
2069 |
|
2070 |
For each of the images that you wish the script to replace, you must have a |
2071 |
high-resolution variant in the same folder named correctly and it will be |
2072 |
detected automatically. For example if your image is in <my_directory> and is |
2073 |
named "my_image.jpg" the script will look in the same directory for an image |
2074 |
named "my_image@2x.jpg". |
2075 |
|
2076 |
In case that you don’t want to use a high-resolution image, the |
2077 |
normal-resolution image will be visible. |
2078 |
|
2079 |
More branding |
2080 |
~~~~~~~~~~~~~ |
2081 |
|
2082 |
Although, it is not 100% branding-related, further verbal customization is |
2083 |
feasible. |
2084 |
|
2085 |
**EMAILS** |
2086 |
|
2087 |
The output of all email `*`.txt files will be already customized to contain your |
2088 |
company and service names but you can further alter their content if you feel it |
2089 |
best fits your needs. |
2090 |
|
2091 |
In order to overwrite one or more email-templates you need to place your |
2092 |
modified <email-file>.txt files respecting the following structure: |
2093 |
|
2094 |
**/etc/synnefo/templates/** |
2095 |
**im/** |
2096 |
| activation_email.txt |
2097 |
| email.txt |
2098 |
| invitation.txt |
2099 |
| switch_accounts_email.txt |
2100 |
| welcome_email.txt |
2101 |
**projects/** |
2102 |
| project_approval_notification.txt |
2103 |
| project_denial_notification.txt |
2104 |
| project_membership_change_notification.txt |
2105 |
| project_membership_enroll_notification.txt |
2106 |
| project_membership_leave_request_notification.txt |
2107 |
| project_membership_request_notification.txt |
2108 |
| project_suspension_notification.txt |
2109 |
| project_termination_notification.txt |
2110 |
**registration/** |
2111 |
| email_change_email.txt |
2112 |
| password_email.txt |
2113 |
|
2114 |
Feel free to omit any of the above files you do not wish to overwrite. |
2115 |
|
2116 |
Below is a list of all emails sent by Synnefo to users along with a short |
2117 |
description and a link to their content: |
2118 |
|
2119 |
* ``snf-astakos-app/astakos/im/templates/im/email.txt`` |
2120 |
Base email template. Contains a contact email and a “thank you” message. |
2121 |
(`Link <https://code.grnet.gr/projects/synnefo/repository/revisions/master/changes/snf-astakos-app/astakos/im/templates/im/email.txt>`_) |
2122 |
* ``snf-astakos-app/astakos/im/templates/im/activation_email.txt`` Email sent to |
2123 |
user that prompts him/her to click on a link provided to activate the account. |
2124 |
Extends “email.txt” (`Link <https://code.grnet.gr/projects/synnefo/repository/revisions/master/changes/snf-astakos-app/astakos/im/templates/im/activation_email.txt>`_) |
2125 |
* ``snf-astakos-app/astakos/im/templates/im/invitation.txt`` Email sent to an |
2126 |
invited user. He/she has to click on a link provided to activate the account. |
2127 |
Extends “email.txt” (`Link <https://code.grnet.gr/projects/synnefo/repository/revisions/master/changes/snf-astakos-app/astakos/im/templates/im/invitation.txt>`_) |
2128 |
* ``snf-astakos-app/astakos/im/templates/im/switch_accounts_email.txt`` Email |
2129 |
sent to user upon his/her request to associate this email address with a |
2130 |
shibboleth account. He/she has to click on a link provided to activate the |
2131 |
association. Extends “email.txt” (`Link <https://code.grnet.gr/projects/synnefo/repository/revisions/master/changes/snf-astakos-app/astakos/im/templates/im/switch_accounts_email.txt>`_) |
2132 |
* ``snf-astakos-app/astakos/im/templates/im/welcome_email.txt`` Email sent to |
2133 |
inform the user that his/ her account has been activated. Extends “email.txt” |
2134 |
(`Link <https://code.grnet.gr/projects/synnefo/repository/revisions/master/changes/snf-astakos-app/astakos/im/templates/im/welcome_email.txt>`_) |
2135 |
* ``snf-astakos-app/astakos/im/templates/registration/email_change_email.txt`` |
2136 |
Email sent to user when he/she has requested new email address assignment. The |
2137 |
user has to click on a link provided to validate this action. Extends |
2138 |
“email.txt” (`Link <https://code.grnet.gr/projects/synnefo/repository/revisions/master/changes/snf-astakos-app/astakos/im/templates/registration/email_change_email.txt>`_) |
2139 |
* ``snf-astakos-app/astakos/im/templates/registration/password_email.txt`` Email |
2140 |
sent for resetting password purpose. The user has to click on a link provided |
2141 |
to validate this action. Extends “email.txt” (`Link <https://code.grnet.gr/projects/synnefo/repository/revisions/master/changes/snf-astakos-app/astakos/im/templates/registration/password_email.txt>`_) |
2142 |
* ``snf-astakos-app/astakos/im/templates/im/projects/project_approval_notification.txt`` |
2143 |
Informs the project owner that his/her project has been approved. Extends |
2144 |
“email.txt” (`Link <https://code.grnet.gr/projects/synnefo/repository/revisions/master/changes/snf-astakos-app/astakos/im/templates/im/projects/project_approval_notification.txt>`_) |
2145 |
* ``snf-astakos-app/astakos/im/templates/im/projects/project_denial_notification.txt`` |
2146 |
Informs the project owner that his/her project application has been denied |
2147 |
explaining the reasons. Extends “email.txt” (`Link <https://code.grnet.gr/projects/synnefo/repository/revisions/master/changes/snf-astakos-app/astakos/im/templates/im/projects/project_denial_notification.txt>`_) |
2148 |
* ``snf-astakos-app/astakos/im/templates/im/projects/project_membership_change_notification.txt`` |
2149 |
An email is sent to a user containing information about his project membership |
2150 |
(whether he has been accepted, rejected or removed). Extends “email.txt” (`Link |
2151 |
<https://code.grnet.gr/projects/synnefo/repository/revisions/master/changes/snf-astakos-app/astakos/im/templates/im/projects/project_membership_change_notification.txt>`_) |
2152 |
* ``snf-astakos-app/astakos/im/templates/im/projects/project_membership_enroll_notification.txt`` |
2153 |
Informs a user that he/she has been enrolled to a project. Extends |
2154 |
“email.txt” (`Link <https://code.grnet.gr/projects/synnefo/repository/revisions/master/changes/snf-astakos-app/astakos/im/templates/im/projects/project_membership_enroll_notification.txt>`_) |
2155 |
* ``snf-astakos-app/astakos/im/templates/im/projects/project_membership_leave_request_notification.txt`` |
2156 |
An email is sent to the project owner to make him aware of a user having |
2157 |
requested to leave his project. Extends “email.txt” (`Link <https://code.grnet.gr/projects/synnefo/repository/revisions/master/changes/snf-astakos-app/astakos/im/templates/im/projects/project_membership_leave_request_notification.txt>`_) |
2158 |
* ``snf-astakos-app/astakos/im/templates/im/projects/project_membership_request_notification.txt`` |
2159 |
An email is sent to the project owner to make him/her aware of a user having |
2160 |
requested to join his project. Extends “email.txt” (`Link <https://code.grnet.gr/projects/synnefo/repository/revisions/master/changes/snf-astakos-app/astakos/im/templates/im/projects/project_membership_request_notification.txt>`_) |
2161 |
* ``snf-astakos-app/astakos/im/templates/im/projects/project_suspension_notification.txt`` |
2162 |
An email is sent to the project owner to make him/her aware of his/her project |
2163 |
having been suspended. Extends “email.txt” (`Link <https://code.grnet.gr/projects/synnefo/repository/revisions/master/changes/snf-astakos-app/astakos/im/templates/im/projects/project_suspension_notification.txt>`_) |
2164 |
* ``snf-astakos-app/astakos/im/templates/im/projects/project_termination_notification.txt`` |
2165 |
An email is sent to the project owner to make him/her aware of his/her project |
2166 |
having been terminated. Extends “email.txt” (`Link <https://code.grnet.gr/projects/synnefo/repository/revisions/master/changes/snf-astakos-app/astakos/im/templates/im/projects/project_termination_notification.txt>`_) |
2167 |
|
2168 |
.. warning:: Django templates language: |
2169 |
|
2170 |
If you choose to overwrite these email templates, be mindful of the necessary |
2171 |
information contained in django template variables that must not be omitted, |
2172 |
such as the activation link for activating one’s account and many more. |
2173 |
These variables are contained into {{}} inside the templates. |
2174 |
|
2175 |
**Astakos landing page** |
2176 |
|
2177 |
Astakos generates sensible default values used to display component- |
2178 |
specific details in several places across views (dashboard, cloudbar |
2179 |
etc.). One of these places is Astakos landing page where Synnefo components are |
2180 |
featured. |
2181 |
|
2182 |
In case those values doesn't seem to suit your deployment, Astakos allows |
2183 |
you to override any of them using ``ASTAKOS_COMPONENTS_META`` setting |
2184 |
in your ``/etc/synnefo/20-snf-astakos-app-settings.conf`` configuration file. |
2185 |
|
2186 |
So, for example if you want to add your own image for Astakos service and in the |
2187 |
same time hide Cyclades service from Astakos landing page you can |
2188 |
add the following line to your configuration file: |
2189 |
|
2190 |
.. code-block:: python |
2191 |
|
2192 |
ASTAKOS_COMPONENTS_META = { |
2193 |
'astakos': { |
2194 |
'dashboard': { |
2195 |
'icon': '<path-to-your-icon>' |
2196 |
} |
2197 |
}, |
2198 |
'cyclades': { |
2199 |
'dashboard': { |
2200 |
'show': False |
2201 |
} |
2202 |
} |
2203 |
} |
2204 |
|
2205 |
A complete list of available keys is shown below: |
2206 |
|
2207 |
.. code-block:: python |
2208 |
|
2209 |
'<component-name>' = { |
2210 |
'order': 1, |
2211 |
'dashboard': { |
2212 |
'order': 1, |
2213 |
'show': True, |
2214 |
'description': '<component-description>', |
2215 |
'icon': '<component-icon-path>', |
2216 |
}, |
2217 |
'cloudbar': { |
2218 |
'show': True |
2219 |
} |
2220 |
} |
2221 |
|
2222 |
|
2223 |
**403, 404 and 500 pages** |
2224 |
|
2225 |
Feel free to add your own 403 (HTTP Forbidden), 404 (Page not found) and |
2226 |
500 (server error) pages. |
2227 |
To override the default Synnefo error views, you must write and include any of |
2228 |
the files 403.html, 404.html and 500.html in your |
2229 |
**/etc/synnefo/templates/** directory. |
2230 |
|
2231 |
Their content is up to you, but you may use as guides the default error pages |
2232 |
found in: |
2233 |
|
2234 |
**/synnefo/snf-webproject/synnefo/webproject/templates/** |
2235 |
| 403.html |
2236 |
| 404.html |
2237 |
| 500.html |
2238 |
|
2239 |
|
2240 |
|
2241 |
.. RabbitMQ |
2242 |
|
2243 |
RabbitMQ Broker |
2244 |
--------------- |
2245 |
|
2246 |
Queue nodes run the RabbitMQ sofware, which provides AMQP functionality. To |
2247 |
guarantee high-availability, more than one Queue nodes should be deployed, each |
2248 |
of them belonging to the same `RabbitMQ cluster |
2249 |
<http://www.rabbitmq.com/clustering.html>`_. Synnefo uses the RabbitMQ |
2250 |
active/active `High Available Queues <http://www.rabbitmq.com/ha.html>`_ which |
2251 |
are mirrored between two nodes within a RabbitMQ cluster. |
2252 |
|
2253 |
The RabbitMQ nodes that form the cluster, are declared to Synnefo through the |
2254 |
`AMQP_HOSTS` setting. Each time a Synnefo component needs to connect to |
2255 |
RabbitMQ, one of these nodes is chosen in a random way. The client that Synnefo |
2256 |
uses to connect to RabbitMQ, handles connection failures transparently and |
2257 |
tries to reconnect to a different node. As long as one of these nodes are up |
2258 |
and running, functionality of Synnefo should not be downgraded by the RabbitMQ |
2259 |
node failures. |
2260 |
|
2261 |
All the queues that are being used are declared as durable, meaning that |
2262 |
messages are persistently stored to RabbitMQ, until they get successfully |
2263 |
processed by a client. |
2264 |
|
2265 |
Currently, RabbitMQ is used by the following components: |
2266 |
|
2267 |
* `snf-ganeti-eventd` and `snf-progress-monitor`: |
2268 |
These components send messages concerning the status and progress of |
2269 |
jobs in the Ganeti backend. |
2270 |
* `snf-dispatcher`: This daemon, consumes the messages that are sent from |
2271 |
the above components, and updates the Cyclades DB accordingly. |
2272 |
|
2273 |
|
2274 |
Installation |
2275 |
~~~~~~~~~~~~ |
2276 |
|
2277 |
Please check the RabbitMQ documentation which covers extensively the |
2278 |
`installation of RabbitMQ server <http://www.rabbitmq.com/download.html>`_ and |
2279 |
the setup of a `RabbitMQ cluster <http://www.rabbitmq.com/clustering.html>`_. |
2280 |
Also, check out the `web management plugin |
2281 |
<http://www.rabbitmq.com/management.html>`_ that can be useful for managing and |
2282 |
monitoring RabbitMQ. |
2283 |
|
2284 |
For a basic installation of RabbitMQ on two nodes (node1 and node2) you can do |
2285 |
the following: |
2286 |
|
2287 |
On both nodes, install rabbitmq-server and create a Synnefo user: |
2288 |
|
2289 |
.. code-block:: console |
2290 |
|
2291 |
$ apt-get install rabbitmq-server |
2292 |
$ rabbitmqctl add_user synnefo "example_pass" |
2293 |
$ rabbitmqctl set_permissions synnefo ".*" ".*" ".*" |
2294 |
|
2295 |
Also guarantee that both nodes share the same cookie, by running: |
2296 |
|
2297 |
.. code-block:: console |
2298 |
|
2299 |
$ scp node1:/var/lib/rabbitmq/.erlang.cookie node2:/var/lib/rabbitmq/.erlang.cookie |
2300 |
|
2301 |
and restart the nodes: |
2302 |
|
2303 |
.. code-block:: console |
2304 |
|
2305 |
$ /etc/init.d/rabbitmq-server restart |
2306 |
|
2307 |
|
2308 |
To setup the RabbitMQ cluster run: |
2309 |
|
2310 |
.. code-block:: console |
2311 |
|
2312 |
root@node2: rabbitmqctl stop_app |
2313 |
root@node2: rabbitmqctl reset |
2314 |
root@node2: rabbitmqctl cluster rabbit@node1 rabbit@node2 |
2315 |
root@node2: rabbitmqctl start_app |
2316 |
|
2317 |
You can verify that the cluster is set up correctly by running: |
2318 |
|
2319 |
.. code-block:: console |
2320 |
|
2321 |
root@node2: rabbitmqctl cluster_status |
2322 |
|
2323 |
|
2324 |
Logging |
2325 |
------- |
2326 |
|
2327 |
Logging in Synnefo is using Python's logging module. The module is configured |
2328 |
using dictionary configuration, whose format is described here: |
2329 |
|
2330 |
http://docs.python.org/release/2.7.1/library/logging.html#logging-config-dictschema |
2331 |
|
2332 |
The logging configuration dictionary is defined in |
2333 |
``/etc/synnefo/10-snf-webproject-logging.conf`` |
2334 |
|
2335 |
The administrator can have logging control by modifying the ``LOGGING_SETUP`` |
2336 |
dictionary, and defining subloggers with different handlers and log levels. |
2337 |
|
2338 |
|
2339 |
.. _scale-up: |
2340 |
|
2341 |
Scaling up to multiple nodes |
2342 |
============================ |
2343 |
|
2344 |
Here we will describe how should a large scale Synnefo deployment look like. Make |
2345 |
sure you are familiar with Synnefo and Ganeti before proceeding with this section. |
2346 |
This means you should at least have already set up successfully a working Synnefo |
2347 |
deployment as described in the :ref:`Admin's Installation Guide |
2348 |
<quick-install-admin-guide>` and also read the Administrator's Guide until this |
2349 |
section. |
2350 |
|
2351 |
Graph of a scale-out Synnefo deployment |
2352 |
--------------------------------------- |
2353 |
|
2354 |
Each box in the following graph corresponds to a distinct physical node: |
2355 |
|
2356 |
.. image:: images/synnefo-arch2-roles.png |
2357 |
:width: 100% |
2358 |
:target: _images/synnefo-arch2-roles.png |
2359 |
|
2360 |
The above graph is actually the same with the one at the beginning of this |
2361 |
:ref:`guide <admin-guide>`, with the only difference that here we show the |
2362 |
Synnefo roles of each physical node. These roles are described in the |
2363 |
following section. |
2364 |
|
2365 |
.. _physical-node-roles: |
2366 |
|
2367 |
Physical Node roles |
2368 |
------------------- |
2369 |
|
2370 |
As appears in the previous graph, a scale-out Synnefo deployment consists of |
2371 |
multiple physical nodes that have the following roles: |
2372 |
|
2373 |
* **WEBSERVER**: A web server running in front of gunicorn (e.g.: Apache, nginx) |
2374 |
* **ASTAKOS**: The Astakos application (gunicorn) |
2375 |
* **ASTAKOS_DB**: The Astakos database (postgresql) |
2376 |
* **PITHOS**: The Pithos application (gunicorn) |
2377 |
* **PITHOS_DB**: The Pithos database (postgresql) |
2378 |
* **CYCLADES**: The Cyclades application (gunicorn) |
2379 |
* **CYCLADES_DB**: The Cyclades database (postgresql) |
2380 |
* **MQ**: The message queue (RabbitMQ) |
2381 |
* **GANETI_MASTER**: The Ganeti master of a Ganeti cluster |
2382 |
* **GANETI_NODE** : A VM-capable Ganeti node of a Ganeti cluster |
2383 |
|
2384 |
You will probably also have: |
2385 |
|
2386 |
* **CMS**: The CMS used as a frotend portal for the Synnefo services |
2387 |
* **NS**: A nameserver serving all other Synnefo nodes and resolving Synnefo FQDNs |
2388 |
* **CLIENT**: A machine that runs the Synnefo clients (e.g.: kamaki, Web UI), |
2389 |
most of the times, the end user's local machine |
2390 |
|
2391 |
From this point we will also refer to the following groups of roles: |
2392 |
|
2393 |
* **SYNNEFO**: [ **ASTAKOS**, **ASTAKOS_DB**, **PITHOS**, **PITHOS_DB**, **CYCLADES**, **CYCLADES_DB**, **MQ**, **CMS**] |
2394 |
* **G_BACKEND**: [**GANETI_MASTER**, **GANETI_NODE**] |
2395 |
|
2396 |
Of course, when deploying Synnefo you can combine multiple of the above roles on a |
2397 |
single physical node, but if you are trying to scale out, the above separation |
2398 |
gives you significant advantages. |
2399 |
|
2400 |
So, in the next section we will take a look on what components you will have to |
2401 |
install on each physical node depending on its Synnefo role. We assume the graph's |
2402 |
architecture. |
2403 |
|
2404 |
Components for each role |
2405 |
------------------------ |
2406 |
|
2407 |
When deploying Synnefo in large scale, you need to install different Synnefo |
2408 |
or/and third party components on different physical nodes according to their |
2409 |
Synnefo role, as stated in the previous section. |
2410 |
|
2411 |
Specifically: |
2412 |
|
2413 |
Role **WEBSERVER** |
2414 |
* Synnefo components: `None` |
2415 |
* 3rd party components: Apache |
2416 |
Role **ASTAKOS** |
2417 |
* Synnefo components: `snf-webproject`, `snf-astakos-app` |
2418 |
* 3rd party components: Django, Gunicorn |
2419 |
Role **ASTAKOS_DB** |
2420 |
* Synnefo components: `None` |
2421 |
* 3rd party components: PostgreSQL |
2422 |
Role **PITHOS** |
2423 |
* Synnefo components: `snf-webproject`, `snf-pithos-app`, `snf-pithos-webclient` |
2424 |
* 3rd party components: Django, Gunicorn |
2425 |
Role **PITHOS_DB** |
2426 |
* Synnefo components: `None` |
2427 |
* 3rd party components: PostgreSQL |
2428 |
Role **CYCLADES** |
2429 |
* Synnefo components: `snf-webproject`, `snf-cyclades-app`, `snf-vncauthproxy` |
2430 |
* 3rd party components: Django Gunicorn |
2431 |
Role **CYCLADES_DB** |
2432 |
* Synnefo components: `None` |
2433 |
* 3rd party components: PostgreSQL |
2434 |
Role **MQ** |
2435 |
* Synnefo components: `None` |
2436 |
* 3rd party components: RabbitMQ |
2437 |
Role **GANETI_MASTER** |
2438 |
* Synnefo components: `snf-cyclades-gtools` |
2439 |
* 3rd party components: Ganeti |
2440 |
Role **GANETI_NODE** |
2441 |
* Synnefo components: `snf-cyclades-gtools`, `snf-network`, `snf-image`, `nfdhcpd` |
2442 |
* 3rd party components: Ganeti |
2443 |
Role **CMS** |
2444 |
* Synnefo components: `snf-webproject`, `snf-cloudcms` |
2445 |
* 3rd party components: Django, Gunicorn |
2446 |
Role **NS** |
2447 |
* Synnefo components: `None` |
2448 |
* 3rd party components: BIND |
2449 |
Role **CLIENT** |
2450 |
* Synnefo components: `kamaki`, `snf-image-creator` |
2451 |
* 3rd party components: `None` |
2452 |
|
2453 |
Example scale out installation |
2454 |
------------------------------ |
2455 |
|
2456 |
In this section we describe an example of a medium scale installation which |
2457 |
combines multiple roles on 10 different physical nodes. We also provide a |
2458 |
:ref:`guide <i-synnefo>` to help with such an install. |
2459 |
|
2460 |
We assume that we have the following 10 physical nodes with the corresponding |
2461 |
roles: |
2462 |
|
2463 |
Node1: |
2464 |
**WEBSERVER**, **ASTAKOS** |
2465 |
Guide sections: |
2466 |
* :ref:`apt <i-apt>` |
2467 |
* :ref:`gunicorn <i-gunicorn>` |
2468 |
* :ref:`apache <i-apache>` |
2469 |
* :ref:`snf-webproject <i-webproject>` |
2470 |
* :ref:`snf-astakos-app <i-astakos>` |
2471 |
Node2: |
2472 |
**WEBSERVER**, **PITHOS** |
2473 |
Guide sections: |
2474 |
* :ref:`apt <i-apt>` |
2475 |
* :ref:`gunicorn <i-gunicorn>` |
2476 |
* :ref:`apache <i-apache>` |
2477 |
* :ref:`snf-webproject <i-webproject>` |
2478 |
* :ref:`snf-pithos-app <i-pithos>` |
2479 |
* :ref:`snf-pithos-webclient <i-pithos>` |
2480 |
Node3: |
2481 |
**WEBSERVER**, **CYCLADES** |
2482 |
Guide sections: |
2483 |
* :ref:`apt <i-apt>` |
2484 |
* :ref:`gunicorn <i-gunicorn>` |
2485 |
* :ref:`apache <i-apache>` |
2486 |
* :ref:`snf-webproject <i-webproject>` |
2487 |
* :ref:`snf-cyclades-app <i-cyclades>` |
2488 |
* :ref:`snf-vncauthproxy <i-cyclades>` |
2489 |
Node4: |
2490 |
**WEBSERVER**, **CMS** |
2491 |
Guide sections: |
2492 |
* :ref:`apt <i-apt>` |
2493 |
* :ref:`gunicorn <i-gunicorn>` |
2494 |
* :ref:`apache <i-apache>` |
2495 |
* :ref:`snf-webproject <i-webproject>` |
2496 |
* :ref:`snf-cloudcms <i-cms>` |
2497 |
Node5: |
2498 |
**ASTAKOS_DB**, **PITHOS_DB**, **CYCLADES_DB** |
2499 |
Guide sections: |
2500 |
* :ref:`apt <i-apt>` |
2501 |
* :ref:`postgresql <i-db>` |
2502 |
Node6: |
2503 |
**MQ** |
2504 |
Guide sections: |
2505 |
* :ref:`apt <i-apt>` |
2506 |
* :ref:`rabbitmq <i-mq>` |
2507 |
Node7: |
2508 |
**GANETI_MASTER**, **GANETI_NODE** |
2509 |
Guide sections: |
2510 |
* :ref:`apt <i-apt>` |
2511 |
* :ref:`general <i-backends>` |
2512 |
* :ref:`ganeti <i-ganeti>` |
2513 |
* :ref:`snf-cyclades-gtools <i-gtools>` |
2514 |
* :ref:`snf-network <i-network>` |
2515 |
* :ref:`snf-image <i-image>` |
2516 |
* :ref:`nfdhcpd <i-network>` |
2517 |
Node8: |
2518 |
**GANETI_NODE** |
2519 |
Guide sections: |
2520 |
* :ref:`apt <i-apt>` |
2521 |
* :ref:`general <i-backends>` |
2522 |
* :ref:`ganeti <i-ganeti>` |
2523 |
* :ref:`snf-cyclades-gtools <i-gtools>` |
2524 |
* :ref:`snf-network <i-network>` |
2525 |
* :ref:`snf-image <i-image>` |
2526 |
* :ref:`nfdhcpd <i-network>` |
2527 |
Node9: |
2528 |
**GANETI_NODE** |
2529 |
Guide sections: |
2530 |
`Same as Node8` |
2531 |
Node10: |
2532 |
**GANETI_NODE** |
2533 |
Guide sections: |
2534 |
`Same as Node8` |
2535 |
|
2536 |
All sections: :ref:`Scale out Guide <i-synnefo>` |
2537 |
|
2538 |
|
2539 |
Regions, Zones and Clusters |
2540 |
=========================== |
2541 |
|
2542 |
Region |
2543 |
------ |
2544 |
|
2545 |
A Region is a single Synnefo installation, with |
2546 |
Compute/Network/Image/Volume/Object Store services. A Region is associated with |
2547 |
one set of Synnefo DBs (Astakos DB, Pithos DB and Cyclades DB). Every Region has a |
2548 |
distinct set of API endpoints, e.g., |
2549 |
`https://cloud.example.com/cyclades/compute/v2.0`. Two Regions are most times |
2550 |
located geographically far from each other, e.g. "Europe", "US-East". A Region |
2551 |
comprises multiple Zones. |
2552 |
|
2553 |
Zone |
2554 |
---- |
2555 |
|
2556 |
A Zone is a set of Ganeti clusters, in a potentially geographically distinct |
2557 |
location, e.g. "Athens", "Rome". All clusters have access to the same physical |
2558 |
networks, and are considered a single failure domain, e.g., they access the |
2559 |
network over the same router. A Zone comprises muliple Ganeti clusters. |
2560 |
|
2561 |
Ganeti cluster |
2562 |
-------------- |
2563 |
|
2564 |
A Ganeti cluster is a set of Ganeti nodes (physical machines). One of the nodes |
2565 |
has the role of "Ganeti master". If this node goes down, another node may |
2566 |
undertake the master role. Ganeti nodes run Virtual Machines (VMs). VMs can live |
2567 |
migrate inside a Ganeti cluster. A Ganeti cluster comprises multiple physical |
2568 |
hardware nodes, most times geographically close to each other. |
2569 |
|
2570 |
VM mobility |
2571 |
----------- |
2572 |
|
2573 |
VMs may move across Regions, Zones, Ganeti clusters and physical nodes. Before we |
2574 |
describe how that's possible, we will describe the different kinds of moving, |
2575 |
providing the corresponding terminology: |
2576 |
|
2577 |
Live migration |
2578 |
~~~~~~~~~~~~~~ |
2579 |
|
2580 |
The act of moving a running VM from physical node to physical node without any |
2581 |
impact on its operation. The VM continues to run on its new physical location, |
2582 |
completely unaffected, and without any service downtime or dropped connections. |
2583 |
Live migration typically requires shared storage and networking between the source |
2584 |
and destination nodes. |
2585 |
|
2586 |
Live migration is issued by the administrator in the background and is transparent |
2587 |
to the VM user. |
2588 |
|
2589 |
Failover |
2590 |
~~~~~~~~ |
2591 |
|
2592 |
The act of moving a VM from physical node to physical node by stopping it first on |
2593 |
the source node, then re-starting it on the destination node. There is short |
2594 |
service downtime, during the time the VM boots up, and client connections are |
2595 |
dropped. |
2596 |
|
2597 |
Failover is issued by the administrator in the background and the VM user will |
2598 |
experience a reboot. |
2599 |
|
2600 |
Snapshot Failover |
2601 |
~~~~~~~~~~~~~~~~~ |
2602 |
|
2603 |
The act of moving a VM from physical node to physical node via a point-in-time |
2604 |
snapshot. That is, stopping a VM on the source node, taking a snapshot, then |
2605 |
creating a new VM from that snapshot. |
2606 |
|
2607 |
Snapshot failover is issued by the VM user and not the administrator. |
2608 |
|
2609 |
Disaster Recovery |
2610 |
----------------- |
2611 |
|
2612 |
In Synnefo terminology, Disaster Recovery is the process of sustaining a disaster |
2613 |
in one datacenter, and ensuring business continuity by performing live migration |
2614 |
or failover of running/existing VMs, or respawning VMs from previously made |
2615 |
snapshots. Based on the method used, this can work inside a single Ganeti cluster, |
2616 |
across Ganeti clusters in the same Zone, or across Zones. |
2617 |
|
2618 |
Specifically: |
2619 |
|
2620 |
Live migration is only supported inside a single Ganeti cluster. Ganeti supports |
2621 |
live migration between nodes in the same cluster with or without shared storage. |
2622 |
Live migration is done at the Ganeti level and is transparent to Synnefo. |
2623 |
|
2624 |
Failover is supported inside a Ganeti cluster, across Ganeti clusters and across |
2625 |
Zones. Ganeti supports failover inside a Ganeti cluster with or without shared |
2626 |
storage, which poses minimum downtime for the VM. Failover inside the same Ganeti |
2627 |
cluster is done at the Ganeti level and is transparent to Synnefo. |
2628 |
|
2629 |
Ganeti also provides tools for failing over VMs across different Ganeti clusters, |
2630 |
meaning that one can use them to failover VMs across Ganeti clusters of the same |
2631 |
Zone or across Ganeti clusters of different Zones, thus moving across Zones. |
2632 |
Failing over across different Ganeti clusters requires copying of data, resulting |
2633 |
in longer downtimes, depending on the geographical distance and network between |
2634 |
them. Failover across Ganeti clusters, either in the same or different Zones, is |
2635 |
not transparent to Synnefo and requires manual import of intances at Synnefo level |
2636 |
too, by the administrator. |
2637 |
|
2638 |
Snapshot failover supports moving VMs across all domains. It is issued by the VM |
2639 |
user and is done at the Synnefo level without the need of running anything at the |
2640 |
Ganeti level or by the administrator. |
2641 |
|
2642 |
In the future Synnefo will also support moving VMs across different Regions. |
2643 |
|
2644 |
|
2645 |
Upgrade Notes |
2646 |
============= |
2647 |
|
2648 |
.. toctree:: |
2649 |
:maxdepth: 1 |
2650 |
|
2651 |
v0.12 -> v0.13 <upgrade/upgrade-0.13> |
2652 |
v0.13 -> v0.14 <upgrade/upgrade-0.14> |
2653 |
v0.14 -> v0.14.2 <upgrade/upgrade-0.14.2> |
2654 |
v0.14.5 -> v0.14.6 <upgrade/upgrade-0.14.6> |
2655 |
v0.14.7 -> v0.14.8 <upgrade/upgrade-0.14.8> |
2656 |
v0.14.9 -> v0.14.10 <upgrade/upgrade-0.14.10> |
2657 |
v0.14 -> v0.15 <upgrade/upgrade-0.15> |
2658 |
|
2659 |
|
2660 |
Changelog, NEWS |
2661 |
=============== |
2662 |
|
2663 |
|
2664 |
* v0.14.10 :ref:`Changelog <Changelog-0.14.10>`, :ref:`NEWS <NEWS-0.14.10>` |
2665 |
* v0.14.9 :ref:`Changelog <Changelog-0.14.9>`, :ref:`NEWS <NEWS-0.14.9>` |
2666 |
* v0.14.8 :ref:`Changelog <Changelog-0.14.8>`, :ref:`NEWS <NEWS-0.14.8>` |
2667 |
* v0.14.7 :ref:`Changelog <Changelog-0.14.7>`, :ref:`NEWS <NEWS-0.14.7>` |
2668 |
* v0.14.6 :ref:`Changelog <Changelog-0.14.6>`, :ref:`NEWS <NEWS-0.14.6>` |
2669 |
* v0.14.5 :ref:`Changelog <Changelog-0.14.5>`, :ref:`NEWS <NEWS-0.14.5>` |
2670 |
* v0.14.4 :ref:`Changelog <Changelog-0.14.4>`, :ref:`NEWS <NEWS-0.14.4>` |
2671 |
* v0.14.3 :ref:`Changelog <Changelog-0.14.3>`, :ref:`NEWS <NEWS-0.14.3>` |
2672 |
* v0.14.2 :ref:`Changelog <Changelog-0.14.2>`, :ref:`NEWS <NEWS-0.14.2>` |
2673 |
* v0.14 :ref:`Changelog <Changelog-0.14>`, :ref:`NEWS <NEWS-0.14>` |
2674 |
* v0.13 :ref:`Changelog <Changelog-0.13>`, :ref:`NEWS <NEWS-0.13>` |