Synnefo

.. _admin-guide:

Synnefo Administrator's Guide

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

This is the complete Synnefo Administrator's Guide.

.. _syn+archip:

General Synnefo Architecture

============================

The following figure shows a detailed view of the whole Synnefo architecture

and how it interacts with multiple Ganeti clusters. We hope that after reading

the Administrator's Guide you will be able to understand every component and

all the interactions between them.

.. image:: images/synnefo-arch2.png

   :width: 100%

   :target: _images/synnefo-arch2.png

Synnefo also supports RADOS as an alternative storage backend for

Files/Images/VM disks. You will find the :ref:`corresponding figure

<syn+archip+rados>` later in this guide.

Identity Service (Astakos)

==========================

Authentication methods

----------------------

Astakos supports multiple authentication methods:

 * local username/password

 * LDAP / Active Directory

 * SAML 2.0 (Shibboleth) federated logins

 * Google

 * Twitter

 * LinkedIn

.. _shibboleth-auth:

Shibboleth Authentication

~~~~~~~~~~~~~~~~~~~~~~~~~

Astakos can delegate user authentication to a Shibboleth federation.

To setup shibboleth, install package::

  apt-get install libapache2-mod-shib2

Change appropriately the configuration files in ``/etc/shibboleth``.

Add in ``/etc/apache2/sites-available/synnefo-ssl``::

  ShibConfig /etc/shibboleth/shibboleth2.xml

  Alias      /shibboleth-sp /usr/share/shibboleth

  <Location /ui/login/shibboleth>

    AuthType shibboleth

    ShibRequireSession On

    ShibUseHeaders On

    require valid-user

  </Location>

and before the line containing::

  ProxyPass        / http://localhost:8080/ retry=0

add::

  ProxyPass /Shibboleth.sso !

Then, enable the shibboleth module::

  a2enmod shib2

After passing through the apache module, the following tokens should be

available at the destination::

  eppn # eduPersonPrincipalName

  Shib-InetOrgPerson-givenName

  Shib-Person-surname

  Shib-Person-commonName

  Shib-InetOrgPerson-displayName

  Shib-EP-Affiliation

  Shib-Session-ID

Finally, add 'shibboleth' in ``ASTAKOS_IM_MODULES`` list. The variable resides

inside the file ``/etc/synnefo/20-snf-astakos-app-settings.conf``

Twitter Authentication

~~~~~~~~~~~~~~~~~~~~~~

To enable twitter authentication while signed in under a Twitter account,

visit dev.twitter.com/apps.

Click Create an application.

Fill the necessary information and for callback URL give::

    https://node1.example.com/ui/login/twitter/authenticated

Finally, add 'twitter' in ``ASTAKOS_IM_MODULES`` list. The variable resides

inside the file ``/etc/synnefo/20-snf-astakos-app-settings.conf``

Google Authentication

~~~~~~~~~~~~~~~~~~~~~

To enable google authentication while signed in under a Google account,

visit https://code.google.com/apis/console/.

Under API Access select Create another client ID, select Web application,

expand more options in Your site or hostname section and in Authorized

Redirect URIs add:

Fill the necessary information and for callback URL give::

    https://node1.example.com/ui/login/google/authenticated

Finally, add 'google' in ``ASTAKOS_IM_MODULES`` list. The variable resides

inside the file ``/etc/synnefo/20-snf-astakos-app-settings.conf``

Working with Astakos

--------------------

User registration

~~~~~~~~~~~~~~~~~

When a new user signs up, he/she is not directly marked as active. You can see 

his/her state by running (on the machine that runs the Astakos app):

.. code-block:: console

   $ snf-manage user-list

More detailed user status is provided in the `status` field of the `user-show` 

command:

.. code-block:: console

  $ snf-manage user-show <user-id>

  id                  : 6

  uuid                : 78661411-5eed-412f-a9ea-2de24f542c2e

  status              : Accepted/Active (accepted policy: manual)

  email               : user@synnefo.org

  ....

Based on the `astakos-app` configuration, there are several ways for a user to

get verified and activated in order to be able to login. We discuss the user

verification and activation flow in the following section.

User activation flow

````````````````````

A user can register for an account using the astakos signup form. Once the form

is submited successfully a user entry is created in astakos database. That entry

is passed through the astakos activation backend which handles whether the user

should be automatically verified and activated.

Email verification

``````````````````

The verification process takes place in order to ensure that the user owns the

email provided during the signup process. By default, after each successful

signup astakos notifies user with an verification url via email. 

At this stage:

    * subsequent registrations invalidate and delete the previous registrations 

      of the same email address.

    * in case user misses the initial notification, additional emails can be

      send either via the url which is prompted to the user if he tries to

      login, or by the administrator using the ``snf-manage user-activation-send

      <userid>`` command.

    * administrator may also enforce a user to get verified using the

      ``snf-manage user-modify --verify <userid>`` command.

Account activation

``````````````````

Once the user gets verified, it is time for Astakos to decide whether or not to

proceed through user activation process. If ``ASTAKOS_MODERATION_ENABLED``

setting is set to ``False`` (default value) user gets activated automatically. 

In case the moderation is enabled Astakos may still automatically activate the

user in the following cases:

    * User email matches any of the regular expressions defined in

      ``ASTAKOS_RE_USER_EMAIL_PATTERNS`` (defaults to ``[]``)

    * User used a signup method (e.g. ``shibboleth``) for which automatic

      activation is enabled (see 

      :ref:`authentication methods policies <auth_methods_policies>`).

If all of the above fail to trigger automatic activation, an email is sent to

the persons listed in ``HELPDESK``, ``MANAGERS`` and ``ADMINS`` settings,

notifing that there is a new user pending for moderation and that it's up to

the administrator to decide if the user should be activated. The UI also shows

a corresponding 'pending moderation' message to the user. The administrator can

activate a user using the ``snf-manage user-modify`` command:

.. code-block:: console

    # command to activate a pending user

    $ snf-manage user-modify --accept <userid>

    # command to reject a pending user

    $ snf-manage user-modify --reject --reject-reason="spammer" <userid>

Once the activation process finishes, a greeting message is sent to the user

email address and a notification for the activation to the persons listed in

``HELPDESK``, ``MANAGERS`` and ``ADMINS`` settings. Once activated the user is

able to login and access the Synnefo services.

Additional authentication methods

`````````````````````````````````

Astakos supports third party logins from external identity providers. This

can be usefull since it allows users to use their existing credentials to 

login to astakos service.

Currently astakos supports the following identity providers:

    * `Shibboleth <http://www.internet2.edu/shibboleth>`_ (module name

      ``shibboleth``)

    * `Google <https://developers.google.com/accounts/docs/OAuth2>`_ (module

      name ``google``)

    * `Twitter <https://dev.twitter.com/docs/auth>`_ (module name ``twitter``)

    * `LinkedIn <http://developer.linkedin.com/documents/authentication>`_

      (module name ``linkedin``)

To enable any of the above modules (by default only ``local`` accounts are

allowed), retrieve and set the required provider settings and append the 

module name in ``ASTAKOS_IM_MODULES``.

.. code-block:: python

    # settings from https://code.google.com/apis/console/

    ASTAKOS_GOOGLE_CLIENT_ID = '1111111111-epi60tvimgha63qqnjo40cljkojcann3.apps.googleusercontent.com'

    ASTAKOS_GOOGLE_SECRET = 'tNDQqTDKlTf7_LaeUcWTWwZM'

    # let users signup and login using their google account

    ASTAKOS_IM_MODULES = ['local', 'google']

.. _auth_methods_policies:

Authentication method policies

``````````````````````````````

Astakos allows you to override the default policies for each enabled provider 

separately by adding the approriate settings in your ``.conf`` files in the 

following format:

**ASTAKOS_AUTH_PROVIDER_<module>_<policy>_POLICY**

Available policies are:

    * **CREATE** Users can signup using that provider (default: ``True``) 

    * **REMOVE/ADD** Users can remove/add login method from their profile 

      (default: ``True``)

    * **AUTOMODERATE** Automatically activate users that signup using that

      provider (default: ``False``)

    * **LOGIN** Whether or not users can use the provider to login (default:

      ``True``).

e.g. to enable automatic activation for your academic users, while keeping 

locally signed up users under moderation you can apply the following settings.

.. code-block:: python

    ASTAKOS_AUTH_PROVIDER_SHIBBOLETH_AUTOMODERATE_POLICY = True

    ASTAKOS_AUTH_PROVIDER_SHIBBOLETH_REMOVE_POLICY = False

User login

~~~~~~~~~~

During the logging procedure, the user is authenticated by the respective

identity provider.

If ``ASTAKOS_RECAPTCHA_ENABLED`` is set and the user fails several times

(``ASTAKOS_RATELIMIT_RETRIES_ALLOWED`` setting) to provide the correct

credentials for a local account, he/she is then prompted to solve a captcha

challenge.

Upon success, the system renews the token (if it has expired), logins the user

and sets the cookie, before redirecting the user to the ``next`` parameter

value.

Setting quota limits

~~~~~~~~~~~~~~~~~~~~

Set default quota

`````````````````

In 20-snf-astakos-app-settings.conf, 

uncomment the default setting ``ASTAKOS_SERVICES``

and customize the ``'uplimit'`` values.

These are the default base quota for all users.

To apply your configuration run::

    # snf-manage astakos-init --load-service-resources

    # snf-manage quota --sync

Set base quota for individual users

```````````````````````````````````

For individual users that need different quota than the default

you can set it for each resource like this::

    # use this to display quota / uuid

    # snf-manage user-show 'uuid or email' --quota

    # snf-manage user-modify 'user-uuid' --set-base-quota 'cyclades.vm' 10

Enable the Projects feature

~~~~~~~~~~~~~~~~~~~~~~~~~~~

If you want to enable the projects feature so that users may apply

on their own for resources by creating and joining projects,

in ``20-snf-astakos-app-settings.conf`` set::

    # this will make the 'projects' page visible in the dashboard

    ASTAKOS_PROJECTS_VISIBLE = True

You can change the maximum allowed number of pending project applications

per user with::

    # snf-manage resource-modify astakos.pending_app --limit <number>

You can also set a user-specific limit with::

    # snf-manage user-modify 'user-uuid' --set-base-quota 'astakos.pending_app' 5

When users apply for projects they are not automatically granted

the resources. They must first be approved by the administrator.

To list pending project applications in astakos::

    # snf-manage project-list --pending

Note the last column, the application id. To approve it::

    # <app id> from the last column of project-list

    # snf-manage project-control --approve <app id>

To deny an application::

    # snf-manage project-control --deny <app id>

Users designated as *project admins* can approve, deny, or modify

an application through the web interface. In

``20-snf-astakos-app-settings.conf`` set::

    # UUIDs of users that can approve or deny project applications from the web.

    ASTAKOS_PROJECT_ADMINS = [<uuid>, ...]

Astakos advanced operations

---------------------------

Adding "Terms of Use"

~~~~~~~~~~~~~~~~~~~~~

Astakos supports versioned terms-of-use. First of all you need to create an

html file that will contain your terms. For example, create the file

``/usr/share/synnefo/sample-terms.html``, which contains the following:

.. code-block:: console

   <h1>My cloud service terms</h1>

   These are the example terms for my cloud service

Then, add those terms-of-use with the snf-manage command:

.. code-block:: console

   $ snf-manage term-add /usr/share/synnefo/sample-terms.html

Your terms have been successfully added and you will see the corresponding link

appearing in the Astakos web pages' footer.

During the account registration, if there are approval terms, the user is

presented with an "I agree with the Terms" checkbox that needs to get checked

in order to proceed.

In case there are new approval terms that the user has not signed yet, the

``signed_terms_required`` view decorator redirects to the ``approval_terms``

view, so the user will be presented with the new terms the next time he/she

logins.

Enabling reCAPTCHA

~~~~~~~~~~~~~~~~~~

Astakos supports the `reCAPTCHA <http://www.google.com/recaptcha>`_ feature.

If enabled, it protects the Astakos forms from bots. To enable the feature, go

to https://www.google.com/recaptcha/admin/create and create your own reCAPTCHA

key pair. Then edit ``/etc/synnefo/20-snf-astakos-app-settings.conf`` and set

the corresponding variables to reflect your newly created key pair. Finally, set

the ``ASTAKOS_RECAPTCHA_ENABLED`` variable to ``True``:

.. code-block:: console

   ASTAKOS_RECAPTCHA_PUBLIC_KEY = 'example_recaptcha_public_key!@#$%^&*('

   ASTAKOS_RECAPTCHA_PRIVATE_KEY = 'example_recaptcha_private_key!@#$%^&*('

   ASTAKOS_RECAPTCHA_ENABLED = True

Restart the service on the Astakos node(s) and you are ready:

.. code-block:: console

   # /etc/init.d/gunicorn restart

Checkout your new Sign up page. If you see the reCAPTCHA box, you have setup

everything correctly.

Astakos internals

-----------------

X-Auth-Token

~~~~~~~~~~~~

Alice requests a specific resource from a cloud service e.g.: Pithos. In the

request she supplies the `X-Auth-Token` to identify whether she is eligible to

perform the specific task. The service contacts Astakos through its

``/account/v1.0/authenticate`` api call (see :ref:`authenticate-api-label`)

providing the specific ``X-Auth-Token``. Astakos checkes whether the token

belongs to an active user and it has not expired and returns a dictionary

containing user related information. Finally the service uses the ``uniq``

field included in the dictionary as the account string to identify the user

accessible resources.

.. _authentication-label:

Django Auth methods and Backends

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Astakos incorporates Django user authentication system and extends its User model.

Since username field of django User model has a limitation of 30 characters,

AstakosUser is **uniquely** identified by the ``email`` instead. Therefore,

``astakos.im.authentication_backends.EmailBackend`` is served to authenticate a

user using email if the first argument is actually an email, otherwise tries

the username.

A new AstakosUser instance is assigned with a uui as username and also with a

``auth_token`` used by the cloud services to authenticate the user.

``astakos.im.authentication_backends.TokenBackend`` is also specified in order

to authenticate the user using the email and the token fields.

Logged on users can perform a number of actions:

 * access and edit their profile via: ``/im/profile``.

 * change their password via: ``/im/password``

 * send feedback for grnet services via: ``/im/send_feedback``

 * logout (and delete cookie) via: ``/im/logout``

Internal Astakos requests are handled using cookie-based Django user sessions.

External systems should forward to the ``/login`` URI. The server,

depending on its configuration will redirect to the appropriate login page.

When done with logging in, the service's login URI should redirect to the URI

provided with next, adding user and token parameters, which contain the email

and token fields respectively.

The login URI accepts the following parameters:

======================  =========================

Request Parameter Name  Value

======================  =========================

next                    The URI to redirect to when the process is finished

renew                   Force token renewal (no value parameter)

force                   Force logout current user (no value parameter)

======================  =========================

External systems inside the ``ASTAKOS_COOKIE_DOMAIN`` scope can acquire the

user information by the cookie identified by ``ASTAKOS_COOKIE_NAME`` setting

(set during the login procedure).

Finally, backend systems having acquired a token can use the

:ref:`authenticate-api-label` API call from a private network or through HTTPS.

Compute/Network/Image Service (Cyclades)

========================================

Working with Cyclades

---------------------

Managing Ganeti Backends

~~~~~~~~~~~~~~~~~~~~~~~~

Since v0.11, Synnefo is able to manage multiple Ganeti clusters (backends)

making it capable to scale linearly to tens of thousands of VMs. Backends

can be dynamically added or removed via `snf-manage` commands.

Each newly created VM is allocated to a Ganeti backend by the Cyclades backend

allocator. The VM is "pinned" to this backend, and can not change through its

lifetime. The backend allocator decides in which backend to spawn the VM based

on the available resources of each backend, trying to balance the load between

them.

Handling of Networks, as far as backends are concerned, is based on whether the

network is public or not. Public networks are created through the `snf-manage

network-create` command, and are only created on one backend. Private networks

are created on all backends, in order to ensure that VMs residing on different

backends can be connected to the same private network.

Listing existing backends

`````````````````````````

To list all the Ganeti backends known to Synnefo, we run:

.. code-block:: console

   $ snf-manage backend-list

Adding a new Ganeti backend

```````````````````````````

Backends are dynamically added under the control of Synnefo with `snf-manage

backend-add` command. In this section it is assumed that a Ganeti cluster,

named ``cluster.example.com`` is already up and running and configured to be

able to host Synnefo VMs.

To add this Ganeti cluster, we run:

.. code-block:: console

   $ snf-manage backend-add --clustername=cluster.example.com --user="synnefo_user" --pass="synnefo_pass"

where ``clustername`` is the Cluster hostname of the Ganeti cluster, and

``user`` and ``pass`` are the credentials for the `Ganeti RAPI user

<http://docs.ganeti.org/ganeti/2.2/html/rapi.html#users-and-passwords>`_.  All

backend attributes can be also changed dynamically using the `snf-manage

backend-modify` command.

``snf-manage backend-add`` will also create all existing private networks to

the new backend. You can verify that the backend is added, by running

`snf-manage backend-list`.

Note that no VMs will be spawned to this backend, since by default it is in a

``drained`` state after addition and also it has no public network assigned to

it.

So, first you need to create its public network, make sure everything works as

expected and finally make it active by un-setting the ``drained`` flag. You can

do this by running:

.. code-block:: console

   $ snf-manage backend-modify --drained=False <backend_id>

Removing an existing Ganeti backend

```````````````````````````````````

In order to remove an existing backend from Synnefo, we run:

.. code-block:: console

   # snf-manage backend-remove <backend_id>

This command will fail if there are active VMs on the backend. Also, the

backend is not cleaned before removal, so all the Synnefo private networks

will be left on the Ganeti nodes. You need to remove them manually.

Allocation of VMs in Ganeti backends

````````````````````````````````````

As already mentioned, the Cyclades backend allocator is responsible for

allocating new VMs to backends. This allocator does not choose the exact Ganeti

node that will host the VM but just the Ganeti backend. The exact node is

chosen by the Ganeti cluster's allocator (hail).

The decision about which backend will host a VM is based on the available

resources. The allocator computes a score for each backend, that shows its load

factor, and the one with the minimum score is chosen. The admin can exclude

backends from the allocation phase by marking them as ``drained`` by running:

.. code-block:: console

   $ snf-manage backend-modify --drained=True <backend_id>

The backend resources are periodically updated, at a period defined by

the ``BACKEND_REFRESH_MIN`` setting, or by running `snf-manage backend-update-status`

command. It is advised to have a cron job running this command at a smaller

interval than ``BACKEND_REFRESH_MIN`` in order to remove the load of refreshing

the backends stats from the VM creation phase.

Finally, the admin can decide to have a user's VMs being allocated to a

specific backend, with the ``BACKEND_PER_USER`` setting. This is a mapping

between users and backends. If the user is found in ``BACKEND_PER_USER``, then

Synnefo allocates all his/hers VMs to the specific backend in the variable,

even if is marked as drained (useful for testing).

Allocation based on disk-templates

**********************************

Besides the available resources of each Ganeti backend, the allocator takes

into consideration the disk template of the instance when trying to allocate it

to a Ganeti backend. Specifically, the allocator checks if the flavor of the

instance belongs to the available disk templates of each Ganeti backend.

A Ganeti cluster has a list of enabled disk templates

(`--enabled-disk-templates`) and a list of allowed disk templates for new

instances (`--ipolicy-disk-templates`). See the `gnt-cluster` manpage for more

details about these options.

When Synnefo allocates an instance, it checks whether the disk template of the

new instance belongs both in the enabled and ipolicy disk templates. You can

see the list of the available disk-templates by running `snf-manage

backend-list`. This list should be updated automatically after changing

these options in Ganeti and it can also be updated by running `snf-manage

backend-update-status`.

So the administrator, can route instances on different backends based on their

flavor disk template, by modifying the enabled or ipolicy disk templates of

each backend.  Also, the administrator can route instances between different

nodes of the same Ganeti backend, by modifying the same options at the

nodegroup level (see `gnt-group` manpage for mor details).

Managing Virtual Machines

~~~~~~~~~~~~~~~~~~~~~~~~~

As mentioned, Cyclades uses Ganeti for management of VMs. The administrator can

handle Cyclades VMs just like any other Ganeti instance, via `gnt-instance`

commands. All Ganeti instances that belong to Synnefo, are separated from

others, by a prefix in their names. This prefix is defined in

``BACKEND_PREFIX_ID`` setting in

``/etc/synnefo/20-snf-cyclades-app-backend.conf``.

Apart from handling instances directly in the Ganeti level, a number of `snf-manage`

commands are available:

* ``snf-manage server-list``: List servers

* ``snf-manage server-show``: Show information about a server in the Cyclades DB

* ``snf-manage server-inspect``: Inspect the state of a server both in DB and Ganeti

* ``snf-manage server-modify``: Modify the state of a server in the Cycldes DB

* ``snf-manage server-create``: Create a new server

* ``snf-manage server-import``: Import an existing Ganeti instance to Cyclades

Managing Virtual Networks

~~~~~~~~~~~~~~~~~~~~~~~~~

Cyclades is able to create and manage Virtual Networks. Networking is

desployment specific and must be customized based on the specific needs of the

system administrator. For better understanding of networking please refer to

the :ref:`Network <networks>` section.

Exactly as Cyclades VMs can be handled like Ganeti instances, Cyclades Networks

can also by handled as Ganeti networks, via `gnt-network commands`. All Ganeti

networks that belong to Synnefo are named with the prefix

`${BACKEND_PREFIX_ID}-net-`.

There are also the following `snf-manage` commands for managing networks:

* ``snf-manage network-list``: List networks

* ``snf-manage network-show``: Show information about a network in the Cyclades DB

* ``snf-manage network-inspect``: Inspect the state of the network in DB and Ganeti backends

* ``snf-manage network-modify``: Modify the state of a network in the Cycldes DB

* ``snf-manage network-create``: Create a new network

* ``snf-manage network-remove``: Remove an existing network

Managing Network Resources

``````````````````````````

Proper operation of the Cyclades Network Service depends on the unique

assignment of specific resources to each type of virtual network. Specifically,

these resources are:

* IP addresses. Cyclades creates a Pool of IPs for each Network, and assigns a

  unique IP address to each VM, thus connecting it to this Network. You can see

  the IP pool of each network by running `snf-manage network-inspect

  <network_ID>`. IP pools are automatically created and managed by Cyclades,

  depending on the subnet of the Network.

* Bridges corresponding to physical VLANs, which are required for networks of

  type `PRIVATE_PHYSICAL_VLAN`.

* One Bridge corresponding to one physical VLAN which is required for networks of

  type `PRIVATE_MAC_PREFIX`.

IPv4 addresses

**************

An allocation pool of IPv4 addresses is automatically created for every network

that has the attribute `dhcp` set to True. The allocation pool contains the

range of IP addresses that are included in the subnet. The gateway and the

broadcast address of the network are excluded from the allocation pool. The

admin can externally reserve IP addresses to exclude them from automatic

allocation with the `--add-reserved-ips` option of `snf-manage network-modify`

command. For example the following command will reserve two IP addresses

from network with ID `42`:

.. code-block:: console

 snf-manage network-modify --add-reserved-ips=10.0.0.21,10.0.0.22 42

.. warning:: Externally reserving IP addresses is also available at the Ganeti.

 However, when using Cyclades with multiple Ganeti backends, the handling of

 IP pools must be performed from Cyclades!

Bridges

*******

As already mentioned Cyclades use a pool of Bridges that must correspond

to Physical VLAN at the Ganeti level. A bridge from the pool is assigned to

each network of flavor `PHYSICAL_VLAN`. Creation of this pool is done

using `snf-manage pool-create` command. For example the following command

will create a pool containing the brdiges from `prv1` to `prv21`.

.. code-block:: console

   # snf-manage pool-create --type=bridge --base=prv --size=20

You can verify the creation of the pool, and check its contents by running:

.. code-block:: console

   # snf-manage pool-list

   # snf-manage pool-show --type=bridge 1

Finally you can use the `pool-modify` management command in order to externally

reserve the values from pool, extend or shrink the pool if possible.

MAC Prefixes

************

Cyclades also use a pool of MAC prefixes to assign to networks of flavor

`MAC_FILTERED`. Handling of this pool is done exactly as with pool of bridges,

except that the type option must be set to mac-prefix:

.. code-block:: console

   # snf-manage pool-create --type=mac-prefix --base=aa:00:0 --size=65536

The above command will create a pool of MAC prefixes from ``aa:00:1`` to

``b9:ff:f``. The MAC prefix pool is responsible for providing only unicast and

locally administered MAC addresses, so many of these prefixes will be

externally reserved, to exclude from allocation.

Pool reconciliation

*******************

The management command `snf-manage reconcile-pools` can be used that all the

above mentioned pools are consistent and that all values that come from the

pool are not used more than once.

Cyclades advanced operations

----------------------------

Reconciliation mechanism

~~~~~~~~~~~~~~~~~~~~~~~~

On certain occasions, such as a Ganeti or RabbitMQ failure, the state of

Cyclades database may differ from the real state of VMs and networks in the

Ganeti backends. The reconciliation process is designed to synchronize

the state of the Cyclades DB with Ganeti. There are two management commands

for reconciling VMs and Networks

Reconciling Virtual Machines

````````````````````````````

Reconciliation of VMs detects the following conditions:

 * Stale DB servers without corresponding Ganeti instances

 * Orphan Ganeti instances, without corresponding DB entries

 * Out-of-sync state for DB entries wrt to Ganeti instances

To detect all inconsistencies you can just run:

.. code-block:: console

  $ snf-manage reconcile-servers

Adding the `--fix-all` option, will do the actual synchronization:

.. code-block:: console

  $ snf-manage reconcile --fix-all

Please see ``snf-manage reconcile --help`` for all the details.

Reconciling Networks

````````````````````

Reconciliation of Networks detects the following conditions:

  * Stale DB networks without corresponding Ganeti networks

  * Orphan Ganeti networks, without corresponding DB entries

  * Private networks that are not created to all Ganeti backends

  * Unsynchronized IP pools

To detect all inconsistencies you can just run:

.. code-block:: console

  $ snf-manage reconcile-networks

Adding the `--fix-all` option, will do the actual synchronization:

.. code-block:: console

  $ snf-manage reconcile-networks --fix-all

Please see ``snf-manage reconcile-networks --help`` for all the details.

Cyclades internals

------------------

Asynchronous communication with Ganeti backends

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Synnefo uses Google Ganeti backends for VM cluster management. In order for

Cyclades to be able to handle thousands of user requests, Cyclades and Ganeti

communicate asynchronously. Briefly, requests are submitted to Ganeti through

Ganeti's RAPI/HTTP interface, and then asynchronous notifications about the

progress of Ganeti jobs are being created and pushed upwards to Cyclades. The

architecture and communication with a Ganeti backend is shown in the graph

below:

.. image:: images/cyclades-ganeti-communication.png

   :width: 50%

   :target: _images/cyclades-ganeti-communication.png

The Cyclades API server is responsible for handling user requests. Read-only

requests are directly served by looking up the Cyclades DB. If the request

needs an action in the Ganeti backend, Cyclades submit jobs to the Ganeti

master using the `Ganeti RAPI interface

<http://docs.ganeti.org/ganeti/2.2/html/rapi.html>`_.

While Ganeti executes the job, `snf-ganeti-eventd`, `snf-ganeti-hook` and

`snf-progress-monitor` are monitoring the progress of the job and send

corresponding messages to the RabbitMQ servers. These components are part

of `snf-cyclades-gtools` and must be installed on all Ganeti nodes. Specially:

* *snf-ganeti-eventd* sends messages about operations affecting the operating

  state of instances and networks. Works by monitoring the Ganeti job queue.

* *snf-ganeti_hook* sends messages about the NICs of instances. It includes a

  number of `Ganeti hooks <http://docs.ganeti.org/ganeti/2.2/html/hooks.html>`_

  for customisation of operations.

* *snf-progress_monitor* sends messages about the progress of the Image deployment

  phase which is done by the Ganeti OS Definition `snf-image`.

Finally, `snf-dispatcher` consumes messages from the RabbitMQ queues, processes

these messages and properly updates the state of the Cyclades DB. Subsequent

requests to the Cyclades API, will retrieve the updated state from the DB.

Block Storage Service (Archipelago)

===================================

Overview

--------

Archipelago offers Copy-On-Write snapshotable volumes. Pithos images can be used

to provision a volume with Copy-On-Write semantics (i.e. a clone). Snapshots

offer a unique deduplicated image of a volume, that reflects the volume state

during snapshot creation and are indistinguishable from a Pithos image.

Archipelago is used by Cyclades and Ganeti for fast provisioning of VMs based on

CoW volumes. Moreover, it enables live migration of thinly-provisioned VMs with

no physically shared storage.

Archipelago Architecture

------------------------

.. image:: images/archipelago-architecture.png

   :width: 50%

   :target: _images/archipelago-architecture.png

.. _syn+archip+rados:

Overview of Synnefo + Archipelago + RADOS

-----------------------------------------

.. image:: images/synnefo-arch3.png

   :width: 100%

   :target: _images/synnefo-arch3.png

Prereqs

-------

The administrator must initialize the storage backend where archipelago volume

blocks will reside.

In case of a files backend, the administrator must create two directories. One

for the archipelago data blocks and one for the archipelago map blocks. These

should probably be over shared storage to enable sharing archipelago volumes

between multiple nodes. He or she, must also be able to supply a directory where

the pithos data and map blocks reside.

In case of a RADOS backend, the administrator must create two rados pools, one

for data blocks, and one for the map blocks. These pools, must be the same pools

used in pithos, in order to enable volume creation based on pithos images.

Installation

------------

Archipelago consists of

* ``libxseg0``: libxseg used to communicate over shared memory segments

* ``python-xseg``: python bindings for libxseg

* ``archipelago-kernel-dkms``: contains archipelago kernel modules to provide

  block devices to be used as vm disks

* ``python-archipelago``: archipelago python module. Includes archipelago and

  vlmc functionality.

* ``archipelago``: user space tools and peers for the archipelago management and

  volume composition

* ``archipelago-ganeti``: ganeti ext storage scripts, that enable ganeti to

  provision VMs over archipelago

Performing

.. code-block:: console

  $ apt-get install archipelago-ganeti 

should fetch all the required packages and get you up 'n going with archipelago

Bare in mind, that custom librados is required, which is provided in the apt

repo of GRNet.

For now, librados is a dependency of archipelago, even if you do not intend to

use archipelago over RADOS.

Configuration

-------------

Archipelago should work out of the box with a RADOS backend, but basic

configuration can be done in ``/etc/default/archipelago`` .

If you wish to change the storage backend to files, set

.. code-block:: console

   STORAGE="files"

and provide the appropriate settings for files storage backend in the conf file.

These are:

* ``FILED_IMAGES``: directory for archipelago data blocks.

* ``FILED_MAPS``: directory for archipelago map blocks.

* ``PITHOS``: directory of pithos data blocks.