Statistics
| Branch: | Tag: | Revision:

root / docs / admin-guide.rst @ c0d88cf3

History | View | Annotate | Download (39.5 kB)

1
.. _admin-guide:
2

    
3
Synnefo Administrator's Guide
4
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
5

    
6
This is the complete Synnefo Administrator's Guide.
7

    
8

    
9

    
10
General Synnefo Architecture
11
============================
12

    
13
The following graph shows the whole Synnefo architecture and how it interacts
14
with multiple Ganeti clusters. We hope that after reading the Administrator's
15
Guide you will be able to understand every component and all the interactions
16
between them. It is a good idea to first go through the Quick Administrator's
17
Guide before proceeding.
18

    
19
.. image:: images/synnefo-arch2.png
20
   :width: 100%
21
   :target: _images/synnefo-arch2.png
22

    
23

    
24

    
25
Identity Service (Astakos)
26
==========================
27

    
28

    
29
Overview
30
--------
31

    
32
Authentication methods
33
~~~~~~~~~~~~~~~~~~~~~~
34

    
35
Local Authentication
36
````````````````````
37

    
38
LDAP Authentication
39
```````````````````
40

    
41
.. _shibboleth-auth:
42

    
43
Shibboleth Authentication
44
`````````````````````````
45

    
46
Astakos can delegate user authentication to a Shibboleth federation.
47

    
48
To setup shibboleth, install package::
49

    
50
  apt-get install libapache2-mod-shib2
51

    
52
Change appropriately the configuration files in ``/etc/shibboleth``.
53

    
54
Add in ``/etc/apache2/sites-available/synnefo-ssl``::
55

    
56
  ShibConfig /etc/shibboleth/shibboleth2.xml
57
  Alias      /shibboleth-sp /usr/share/shibboleth
58

    
59
  <Location /im/login/shibboleth>
60
    AuthType shibboleth
61
    ShibRequireSession On
62
    ShibUseHeaders On
63
    require valid-user
64
  </Location>
65

    
66
and before the line containing::
67

    
68
  ProxyPass        / http://localhost:8080/ retry=0
69

    
70
add::
71

    
72
  ProxyPass /Shibboleth.sso !
73

    
74
Then, enable the shibboleth module::
75

    
76
  a2enmod shib2
77

    
78
After passing through the apache module, the following tokens should be
79
available at the destination::
80

    
81
  eppn # eduPersonPrincipalName
82
  Shib-InetOrgPerson-givenName
83
  Shib-Person-surname
84
  Shib-Person-commonName
85
  Shib-InetOrgPerson-displayName
86
  Shib-EP-Affiliation
87
  Shib-Session-ID
88

    
89
Finally, add 'shibboleth' in ``ASTAKOS_IM_MODULES`` list. The variable resides
90
inside the file ``/etc/synnefo/20-snf-astakos-app-settings.conf``
91

    
92
Architecture
93
------------
94

    
95
Prereqs
96
-------
97

    
98
Installation
99
------------
100

    
101
Configuration
102
-------------
103

    
104
Working with Astakos
105
--------------------
106

    
107
User activation methods
108
~~~~~~~~~~~~~~~~~~~~~~~
109

    
110
When a new user signs up, he/she is not marked as active. You can see his/her
111
state by running (on the machine that runs the Astakos app):
112

    
113
.. code-block:: console
114

    
115
   $ snf-manage user-list
116

    
117
There are two different ways to activate a new user. Both need access to a
118
running :ref:`mail server <mail-server>`.
119

    
120
Manual activation
121
`````````````````
122

    
123
You can manually activate a new user that has already signed up, by sending
124
him/her an activation email. The email will contain an approriate activation
125
link, which will complete the activation process if followed. You can send the
126
email by running:
127

    
128
.. code-block:: console
129

    
130
   $ snf-manage user-activation-send <user ID or email>
131

    
132
Be sure to have already setup your mail server and defined it in your Synnefo
133
settings, before running the command.
134

    
135
Automatic activation
136
````````````````````
137

    
138
FIXME: Describe Regex activation method
139

    
140
Setting quota limits
141
~~~~~~~~~~~~~~~~~~~~
142

    
143
Set default quotas
144
``````````````````
145

    
146
In 20-snf-astakos-app-settings.conf, 
147
uncomment the default setting ``ASTAKOS_SERVICES``
148
and customize the ``'uplimit'`` values.
149
These are the default base quotas for all users.
150

    
151
To apply your configuration run::
152

    
153
    # snf-manage astakos-init --load-service-resources
154
    # snf-manage astakos-quota --sync
155

    
156
Set base quotas for individual users
157
````````````````````````````````````
158

    
159
For individual users that need different quotas than the default
160
you can set it for each resource like this::
161

    
162
    # use this to display quotas / uuid
163
    # snf-manage user-show 'uuid or email'
164

    
165
    # snf-manage user-set-initial-quota --set-capacity 'user-uuid' 'cyclades.vm' 10
166

    
167
    # this applies the configuration
168
    # snf-manage astakos-quota --sync --user 'user-uuid'
169

    
170

    
171
Enable the Projects feature
172
~~~~~~~~~~~~~~~~~~~~~~~~~~~
173

    
174
If you want to enable the projects feature so that users may apply
175
on their own for resources by creating and joining projects,
176
in ``20-snf-astakos-app-settings.conf`` set::
177

    
178
    # this will allow at most one pending project application per user
179
    ASTAKOS_PENDING_APPLICATION_LIMIT = 1
180
    # this will make the 'projects' page visible in the dashboard
181
    ASTAKOS_PROJECTS_VISIBLE = True
182

    
183
You can specify a user-specific limit on pending project applications
184
with::
185

    
186
    # snf-manage user-update <user id> --max-pending-projects=2
187

    
188
When users apply for projects they are not automatically granted
189
the resources. They must first be approved by the administrator.
190

    
191
To list pending project applications in astakos::
192

    
193
    # snf-manage project-list --pending
194

    
195
Note the last column, the application id. To approve it::
196

    
197
    # <app id> from the last column of project-list
198
    # snf-manage project-control --approve <app id>
199

    
200
To deny an application::
201

    
202
    # snf-manage project-control --deny <app id>
203

    
204
Users designated as *project admins* can approve, deny, or modify
205
an application through the web interface. In
206
``20-snf-astakos-app-settings.conf`` set::
207

    
208
    # UUIDs of users that can approve or deny project applications from the web.
209
    ASTAKOS_PROJECT_ADMINS = [<uuid>, ...]
210

    
211

    
212
Astakos advanced operations
213
---------------------------
214

    
215
Adding "Terms of Use"
216
~~~~~~~~~~~~~~~~~~~~~
217

    
218
Astakos supports versioned terms-of-use. First of all you need to create an
219
html file that will contain your terms. For example, create the file
220
``/usr/share/synnefo/sample-terms.html``, which contains the following:
221

    
222
.. code-block:: console
223

    
224
   <h1>~okeanos terms</h1>
225

    
226
   These are the example terms for ~okeanos
227

    
228
Then, add those terms-of-use with the snf-manage command:
229

    
230
.. code-block:: console
231

    
232
   $ snf-manage term-add /usr/share/synnefo/sample-terms.html
233

    
234
Your terms have been successfully added and you will see the corresponding link
235
appearing in the Astakos web pages' footer.
236

    
237
Enabling reCAPTCHA
238
~~~~~~~~~~~~~~~~~~
239

    
240
Astakos supports the `reCAPTCHA <http://www.google.com/recaptcha>`_ feature.
241
If enabled, it protects the Astakos forms from bots. To enable the feature, go
242
to https://www.google.com/recaptcha/admin/create and create your own reCAPTCHA
243
key pair. Then edit ``/etc/synnefo/20-snf-astakos-app-settings.conf`` and set
244
the corresponding variables to reflect your newly created key pair. Finally, set
245
the ``ASTAKOS_RECAPTCHA_ENABLED`` variable to ``True``:
246

    
247
.. code-block:: console
248

    
249
   ASTAKOS_RECAPTCHA_PUBLIC_KEY = 'example_recaptcha_public_key!@#$%^&*('
250
   ASTAKOS_RECAPTCHA_PRIVATE_KEY = 'example_recaptcha_private_key!@#$%^&*('
251

    
252
   ASTAKOS_RECAPTCHA_ENABLED = True
253

    
254
Restart the service on the Astakos node(s) and you are ready:
255

    
256
.. code-block:: console
257

    
258
   # /etc/init.d/gunicorn restart
259

    
260
Checkout your new Sign up page. If you see the reCAPTCHA box, you have setup
261
everything correctly.
262

    
263

    
264

    
265
File Storage Service (Pithos)
266
=============================
267

    
268
Overview
269
--------
270

    
271
Architecture
272
------------
273

    
274
Prereqs
275
-------
276

    
277
Installation
278
------------
279

    
280
Configuration
281
-------------
282

    
283
Working with Pithos
284
-------------------
285

    
286
Pithos advanced operations
287
--------------------------
288

    
289

    
290

    
291
Compute/Network/Image Service (Cyclades)
292
========================================
293

    
294
Compute Overview
295
----------------
296

    
297
Network Overview
298
----------------
299

    
300
Image Overview
301
--------------
302

    
303
Architecture
304
------------
305

    
306
Asynchronous communication with Ganeti backends
307
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
308
Synnefo uses Google Ganeti backends for VM cluster management. In order for
309
Cyclades to be able to handle thousands of user requests, Cyclades and Ganeti
310
communicate asynchronously. Briefly, requests are submitted to Ganeti through
311
Ganeti's RAPI/HTTP interface, and then asynchronous notifications about the
312
progress of Ganeti jobs are being created and pushed upwards to Cyclades. The
313
architecture and communication with a Ganeti backend is shown in the graph
314
below:
315

    
316
.. image:: images/cyclades-ganeti-communication.png
317
   :width: 50%
318
   :target: _images/cyclades-ganeti-communication.png
319

    
320
The Cyclades API server is responsible for handling user requests. Read-only
321
requests are directly served by looking up the Cyclades DB. If the request
322
needs an action in the Ganeti backend, Cyclades submit jobs to the Ganeti
323
master using the `Ganeti RAPI interface
324
<http://docs.ganeti.org/ganeti/2.2/html/rapi.html>`_.
325

    
326
While Ganeti executes the job, `snf-ganeti-eventd`, `snf-ganeti-hook` and
327
`snf-progress-monitor` are monitoring the progress of the job and send
328
corresponding messages to the RabbitMQ servers. These components are part
329
of `snf-cyclades-gtools` and must be installed on all Ganeti nodes. Specially:
330

    
331
* *snf-ganeti-eventd* sends messages about operations affecting the operating
332
  state of instances and networks. Works by monitoring the Ganeti job queue.
333
* *snf-ganeti_hook* sends messages about the NICs of instances. It includes a
334
  number of `Ganeti hooks <http://docs.ganeti.org/ganeti/2.2/html/hooks.html>`_
335
  for customisation of operations.
336
* *snf-progress_monitor* sends messages about the progress of the Image deployment
337
  phase which is done by the Ganeti OS Definition `snf-image`.
338

    
339
Finally, `snf-dispatcher` consumes messages from the RabbitMQ queues, processes
340
these messages and properly updates the state of the Cyclades DB. Subsequent
341
requests to the Cyclades API, will retrieve the updated state from the DB.
342

    
343

    
344
Prereqs
345
-------
346

    
347
Work in progress. Please refer to :ref:`quick administrator quide <quick-install-admin-guide>`.
348

    
349
Installation
350
------------
351

    
352
Work in progress. Please refer to :ref:`quick administrator quide <quick-install-admin-guide>`.
353

    
354
Configuration
355
-------------
356

    
357
Work in progress. Please refer to :ref:`quick administrator quide <quick-install-admin-guide>`.
358

    
359
Working with Cyclades
360
---------------------
361

    
362
Managing Ganeti Backends
363
~~~~~~~~~~~~~~~~~~~~~~~~
364

    
365
Since v0.11, Synnefo is able to manage multiple Ganeti clusters (backends)
366
making it capable to scale linearly to tens of thousands of VMs. Backends
367
can be dynamically added or removed via `snf-manage` commands.
368

    
369
Each newly created VM is allocated to a Ganeti backend by the Cyclades backend
370
allocator. The VM is "pinned" to this backend, and can not change through its
371
lifetime. The backend allocator decides in which backend to spawn the VM based
372
on the available resources of each backend, trying to balance the load between
373
them.
374

    
375
Handling of Networks, as far as backends are concerned, is based on whether the
376
network is public or not. Public networks are created through the `snf-manage
377
network-create` command, and are only created on one backend. Private networks
378
are created on all backends, in order to ensure that VMs residing on different
379
backends can be connected to the same private network.
380

    
381
Listing existing backends
382
`````````````````````````
383
To list all the Ganeti backends known to Synnefo, we run:
384

    
385
.. code-block:: console
386

    
387
   $ snf-manage backend-list
388

    
389
Adding a new Ganeti backend
390
```````````````````````````
391
Backends are dynamically added under the control of Synnefo with `snf-manage
392
backend-add` command. In this section it is assumed that a Ganeti cluster,
393
named ``cluster.example.com`` is already up and running and configured to be
394
able to host Synnefo VMs.
395

    
396
To add this Ganeti cluster, we run:
397

    
398
.. code-block:: console
399

    
400
   $ snf-manage backend-add --clustername=cluster.example.com --user="synnefo_user" --pass="synnefo_pass"
401

    
402
where ``clustername`` is the Cluster hostname of the Ganeti cluster, and
403
``user`` and ``pass`` are the credentials for the `Ganeti RAPI user
404
<http://docs.ganeti.org/ganeti/2.2/html/rapi.html#users-and-passwords>`_.  All
405
backend attributes can be also changed dynamically using the `snf-manage
406
backend-modify` command.
407

    
408
``snf-manage backend-add`` will also create all existing private networks to
409
the new backend. You can verify that the backend is added, by running
410
`snf-manage backend-list`.
411

    
412
Note that no VMs will be spawned to this backend, since by default it is in a
413
``drained`` state after addition and also it has no public network assigned to
414
it.
415

    
416
So, first you need to create its public network, make sure everything works as
417
expected and finally make it active by un-setting the ``drained`` flag. You can
418
do this by running:
419

    
420
.. code-block:: console
421

    
422
   $ snf-manage backend-modify --drained=False <backend_id>
423

    
424
Removing an existing Ganeti backend
425
```````````````````````````````````
426
In order to remove an existing backend from Synnefo, we run:
427

    
428
.. code-block:: console
429

    
430
   # snf-manage backend-remove <backend_id>
431

    
432
This command will fail if there are active VMs on the backend. Also, the
433
backend is not cleaned before removal, so all the Synnefo private networks
434
will be left on the Ganeti nodes. You need to remove them manually.
435

    
436
Allocation of VMs in Ganeti backends
437
````````````````````````````````````
438
As already mentioned, the Cyclades backend allocator is responsible for
439
allocating new VMs to backends. This allocator does not choose the exact Ganeti
440
node that will host the VM but just the Ganeti backend. The exact node is
441
chosen by the Ganeti cluster's allocator (hail).
442

    
443
The decision about which backend will host a VM is based on the available
444
resources. The allocator computes a score for each backend, that shows its load
445
factor, and the one with the minimum score is chosen. The admin can exclude
446
backends from the allocation phase by marking them as ``drained`` by running:
447

    
448
.. code-block:: console
449

    
450
   $ snf-manage backend-modify --drained=True <backend_id>
451

    
452
The backend resources are periodically updated, at a period defined by
453
the ``BACKEND_REFRESH_MIN`` setting, or by running `snf-manage backend-update-status`
454
command. It is advised to have a cron job running this command at a smaller
455
interval than ``BACKEND_REFRESH_MIN`` in order to remove the load of refreshing
456
the backends stats from the VM creation phase.
457

    
458
Finally, the admin can decide to have a user's VMs being allocated to a
459
specific backend, with the ``BACKEND_PER_USER`` setting. This is a mapping
460
between users and backends. If the user is found in ``BACKEND_PER_USER``, then
461
Synnefo allocates all his/hers VMs to the specific backend in the variable,
462
even if is marked as drained (useful for testing).
463

    
464

    
465
Managing Virtual Machines
466
~~~~~~~~~~~~~~~~~~~~~~~~~
467

    
468
As mentioned, Cyclades uses Ganeti for management of VMs. The administrator can
469
handle Cyclades VMs just like any other Ganeti instance, via `gnt-instance`
470
commands. All Ganeti instances that belong to Synnefo, are separated from
471
others, by a prefix in their names. This prefix is defined in
472
``BACKEND_PREFIX_ID`` setting in
473
``/etc/synnefo/20-snf-cyclades-app-backend.conf``.
474

    
475
Apart from handling instances directly in the Ganeti level, a number of `snf-manage`
476
commands are available:
477

    
478
* ``snf-manage server-list``: List servers
479
* ``snf-manage server-show``: Show information about a server in the Cyclades DB
480
* ``snf-manage server-inspect``: Inspect the state of a server both in DB and Ganeti
481
* ``snf-manage server-modify``: Modify the state of a server in the Cycldes DB
482
* ``snf-manage server-create``: Create a new server
483
* ``snf-manage server-import``: Import an existing Ganeti instance to Cyclades
484

    
485

    
486
Managing Virtual Networks
487
~~~~~~~~~~~~~~~~~~~~~~~~~
488

    
489
Cyclades is able to create and manage Virtual Networks. Networking is
490
desployment specific and must be customized based on the specific needs of the
491
system administrator. For better understanding of networking please refer to
492
the :ref:`Network <networks>` section.
493

    
494
Exactly as Cyclades VMs can be handled like Ganeti instances, Cyclades Networks
495
can also by handled as Ganeti networks, via `gnt-network commands`. All Ganeti
496
networks that belong to Synnefo are named with the prefix
497
`${BACKEND_PREFIX_ID}-net-`.
498

    
499
There are also the following `snf-manage` commands for managing networks:
500

    
501
* ``snf-manage network-list``: List networks
502
* ``snf-manage network-show``: Show information about a network in the Cyclades DB
503
* ``snf-manage network-inspect``: Inspect the state of the network in DB and Ganeti backends
504
* ``snf-manage network-modify``: Modify the state of a network in the Cycldes DB
505
* ``snf-manage network-create``: Create a new network
506
* ``snf-manage network-remove``: Remove an existing network
507

    
508
Managing Network Resources
509
``````````````````````````
510

    
511
Proper operation of the Cyclades Network Service depends on the unique
512
assignment of specific resources to each type of virtual network. Specifically,
513
these resources are:
514

    
515
* IP addresses. Cyclades creates a Pool of IPs for each Network, and assigns a
516
  unique IP address to each VM, thus connecting it to this Network. You can see
517
  the IP pool of each network by running `snf-manage network-inspect
518
  <network_ID>`. IP pools are automatically created and managed by Cyclades,
519
  depending on the subnet of the Network.
520
* Bridges corresponding to physical VLANs, which are required for networks of
521
  type `PRIVATE_PHYSICAL_VLAN`.
522
* One Bridge corresponding to one physical VLAN which is required for networks of
523
  type `PRIVATE_MAC_PREFIX`.
524

    
525
Cyclades allocates those resources from pools that are created by the
526
administrator with the `snf-manage pool-create` management command.
527

    
528
Pool Creation
529
`````````````
530
Pools are created using the `snf-manage pool-create` command:
531

    
532
.. code-block:: console
533

    
534
   # snf-manage pool-create --type=bridge --base=prv --size=20
535

    
536
will create a pool of bridges, containing bridges prv1, prv2,..prv21.
537

    
538
You can verify the creation of the pool, and check its contents by running:
539

    
540
.. code-block:: console
541

    
542
   # snf-manage pool-list
543
   # snf-manage pool-show --type=bridge 1
544

    
545
With the same commands you can handle a pool of MAC prefixes. For example:
546

    
547
.. code-block:: console
548

    
549
   # snf-manage pool-create --type=mac-prefix --base=aa:00:0 --size=65536
550

    
551
will create a pool of MAC prefixes from ``aa:00:1`` to ``b9:ff:f``. The MAC
552
prefix pool is responsible for providing only unicast and locally administered
553
MAC addresses, so many of these prefixes will be externally reserved, to
554
exclude from allocation.
555

    
556
Cyclades advanced operations
557
----------------------------
558

    
559
Reconciliation mechanism
560
~~~~~~~~~~~~~~~~~~~~~~~~
561

    
562
On certain occasions, such as a Ganeti or RabbitMQ failure, the state of
563
Cyclades database may differ from the real state of VMs and networks in the
564
Ganeti backends. The reconciliation process is designed to synchronize
565
the state of the Cyclades DB with Ganeti. There are two management commands
566
for reconciling VMs and Networks
567

    
568
Reconciling Virtual Machines
569
````````````````````````````
570

    
571
Reconciliation of VMs detects the following conditions:
572

    
573
 * Stale DB servers without corresponding Ganeti instances
574
 * Orphan Ganeti instances, without corresponding DB entries
575
 * Out-of-sync state for DB entries wrt to Ganeti instances
576

    
577
To detect all inconsistencies you can just run:
578

    
579
.. code-block:: console
580

    
581
  $ snf-manage reconcile-servers
582

    
583
Adding the `--fix-all` option, will do the actual synchronization:
584

    
585
.. code-block:: console
586

    
587
  $ snf-manage reconcile --fix-all
588

    
589
Please see ``snf-manage reconcile --help`` for all the details.
590

    
591

    
592
Reconciling Networks
593
````````````````````
594

    
595
Reconciliation of Networks detects the following conditions:
596

    
597
  * Stale DB networks without corresponding Ganeti networks
598
  * Orphan Ganeti networks, without corresponding DB entries
599
  * Private networks that are not created to all Ganeti backends
600
  * Unsynchronized IP pools
601

    
602
To detect all inconsistencies you can just run:
603

    
604
.. code-block:: console
605

    
606
  $ snf-manage reconcile-networks
607

    
608
Adding the `--fix-all` option, will do the actual synchronization:
609

    
610
.. code-block:: console
611

    
612
  $ snf-manage reconcile-networks --fix-all
613

    
614
Please see ``snf-manage reconcile-networks --help`` for all the details.
615

    
616

    
617

    
618
Block Storage Service (Archipelago)
619
===================================
620

    
621
Overview
622
--------
623
Archipelago offers Copy-On-Write snapshotable volumes. Pithos images can be used
624
to provision a volume with Copy-On-Write semantics (i.e. a clone). Snapshots
625
offer a unique deduplicated image of a volume, that reflects the volume state
626
during snapshot creation and are indistinguishable from a Pithos image.
627

    
628
Archipelago is used by Cyclades and Ganeti for fast provisioning of VMs based on
629
CoW volumes. Moreover, it enables live migration of thinly-provisioned VMs with
630
no physically shared storage.
631

    
632
Archipelago Architecture
633
------------------------
634

    
635
.. image:: images/archipelago-architecture.png
636
   :width: 50%
637
   :target: _images/archipelago-architecture.png
638

    
639
.. _syn+archip+rados:
640

    
641
Overview of Synnefo + Archipelago + RADOS
642
-----------------------------------------
643

    
644
.. image:: images/synnefo-arch3.png
645
   :width: 100%
646
   :target: _images/synnefo-arch3.png
647

    
648
Prereqs
649
-------
650

    
651
The administrator must initialize the storage backend where archipelago volume
652
blocks will reside.
653

    
654
In case of a files backend, the administrator must create two directories. One
655
for the archipelago data blocks and one for the archipelago map blocks. These
656
should probably be over shared storage to enable sharing archipelago volumes
657
between multiple nodes. He or she, must also be able to supply a directory where
658
the pithos data and map blocks reside.
659

    
660
In case of a RADOS backend, the administrator must create two rados pools, one
661
for data blocks, and one for the map blocks. These pools, must be the same pools
662
used in pithos, in order to enable volume creation based on pithos images.
663

    
664
Installation
665
------------
666

    
667
Archipelago consists of
668

    
669
* ``libxseg0``: libxseg used to communicate over shared memory segments
670
* ``python-xseg``: python bindings for libxseg
671
* ``archipelago-kernel-dkms``: contains archipelago kernel modules to provide
672
  block devices to be used as vm disks
673
* ``python-archipelago``: archipelago python module. Includes archipelago and
674
  vlmc functionality.
675
* ``archipelago``: user space tools and peers for the archipelago management and
676
  volume composition
677
* ``archipelago-ganeti``: ganeti ext storage scripts, that enable ganeti to
678
  provision VMs over archipelago
679

    
680
Performing
681

    
682
.. code-block:: console
683

    
684
  $ apt-get install archipelago-ganeti 
685

    
686
should fetch all the required packages and get you up 'n going with archipelago
687

    
688
Bare in mind, that custom librados is required, which is provided in the apt
689
repo of GRNet.
690

    
691

    
692
For now, librados is a dependency of archipelago, even if you do not intend to
693
use archipelago over RADOS.
694

    
695
Configuration
696
-------------
697
Archipelago should work out of the box with a RADOS backend, but basic
698
configuration can be done in ``/etc/default/archipelago`` .
699

    
700
If you wish to change the storage backend to files, set
701

    
702
.. code-block:: console
703

    
704
   STORAGE="files"
705

    
706
and provide the appropriate settings for files storage backend in the conf file.
707

    
708
These are:
709

    
710
* ``FILED_IMAGES``: directory for archipelago data blocks.
711
* ``FILED_MAPS``: directory for archipelago map blocks.
712
* ``PITHOS``: directory of pithos data blocks.
713
* ``PITHOSMAPS``: directory of pithos map blocks.
714

    
715
The settings for RADOS storage backend are:
716

    
717
* ``RADOS_POOL_MAPS``: The pool where archipelago and pithos map blocks reside.
718
* ``RADOS_POOL_BLOCKS``: The pool where archipelago and pithos data blocks
719
  reside.
720

    
721
Examples can be found in the conf file.
722

    
723
Be aware that archipelago infrastructure doesn't provide default values for this
724
settings. If they are not set in the conf file, archipelago will not be able to
725
function.
726

    
727
Archipelago also provides ``VERBOSITY`` config options to control the output
728
generated by the userspace peers.
729

    
730
The available options are:
731

    
732
* ``VERBOSITY_BLOCKERB``
733
* ``VERBOSITY_BLOCKERM``
734
* ``VERBOSITY_MAPPER``
735
* ``VERBOSITY_VLMC``
736

    
737
and the available values are:
738

    
739
* 0 : Error only logging.
740
* 1 : Warning logging.
741
* 2 : Info logging.
742
* 3 : Debug logging. WARNING: This options produces tons of output, but the
743
  logrotate daemon should take care of it.
744

    
745
Working with Archipelago
746
------------------------
747

    
748
``archipelago`` provides basic functionality for archipelago.
749

    
750
Usage:
751

    
752
.. code-block:: console
753

    
754
  $ archipelago [-u] command
755

    
756

    
757
Currently it supports the following commands:
758

    
759
* ``start [peer]``
760
  Starts archipelago or the specified peer.
761
* ``stop [peer]``
762
  Stops archipelago or the specified peer.
763
* ``restart [peer]``
764
  Restarts archipelago or the specified peer.
765
* ``status``
766
  Show the status of archipelago.
767

    
768
Available peers: ``blockerm``, ``blockerb``, ``mapperd``, ``vlmcd``.
769

    
770

    
771
``start``, ``stop``, ``restart`` can be combined with the ``-u / --user`` option
772
to affect only the userspace peers supporting archipelago.
773

    
774

    
775

    
776
Archipelago advanced operations
777
-------------------------------
778
The ``vlmc`` tool provides a way to interact with archipelago volumes
779

    
780
* ``vlmc map <volumename>``: maps the volume to a xsegbd device.
781

    
782
* ``vlmc unmap </dev/xsegbd[1-..]>``: unmaps the specified device from the
783
  system.
784

    
785
* ``vlmc create <volumename> --snap <snapname> --size <size>``: creates a new
786
  volume named <volumename> from snapshot name <snapname> with size <size>.
787
  The ``--snap`` and ``--size`` are optional, but at least one of them is
788
  mandatory. e.g:
789

    
790
  ``vlmc create <volumename> --snap <snapname>`` creates a volume named
791
  volumename from snapshot snapname. The size of the volume is the same as
792
  the size of the snapshot.
793

    
794
  ``vlmc create <volumename> --size <size>`` creates an empty volume of size
795
  <size> named <volumename>.
796

    
797
* ``vlmc remove <volumename>``: removes the volume and all the related
798
  archipelago blocks from storage.
799

    
800
* ``vlmc list``: provides a list of archipelago volumes. Currently only works
801
  with RADOS storage backend.
802

    
803
* ``vlmc info <volumename>``: shows volume information. Currently returns only
804
  volume size.
805

    
806
* ``vlmc open <volumename>``: opens an archipelago volume. That is, taking all
807
  the necessary locks and also make the rest of the infrastructure aware of the
808
  operation.
809

    
810
  This operation succeeds if the volume is alread opened.
811

    
812
* ``vlmc close <volumename>``: closes an archipelago volume. That is, performing
813
  all the necessary functions in the insfrastrure to successfully release the
814
  volume. Also releases all the acquired locks.
815

    
816
  ``vlmc close`` should be performed after a ``vlmc open`` operation.
817

    
818
* ``vlmc lock <volumename>``: locks a volume. This step allow the administrator
819
  to lock an archipelago volume, independently from the rest of the
820
  infrastrure.
821

    
822
* ``vlmc unlock [-f] <volumename>``: unlocks a volume. This allow the
823
  administrator to unlock a volume, independently from the rest of the
824
  infrastructure.
825
  The unlock option can be performed only by the blocker that acquired the lock
826
  in the first place. To unlock a volume from another blocker, ``-f`` option
827
  must be used to break the lock.
828

    
829

    
830
The "kamaki" API client
831
=======================
832

    
833
To upload, register or modify an image you will need the **kamaki** tool.
834
Before proceeding make sure that it is configured properly. Verify that
835
*image.url*, *file.url*, *user.url* and *token* are set as needed:
836

    
837
.. code-block:: console
838

    
839
   $ kamaki config list
840

    
841
To chage a setting use ``kamaki config set``:
842

    
843
.. code-block:: console
844

    
845
   $ kamaki config set image.url https://cyclades.example.com/plankton
846
   $ kamaki config set file.url https://pithos.example.com/v1
847
   $ kamaki config set user.url https://accounts.example.com
848
   $ kamaki config set token ...
849

    
850
To test that everything works, try authenticating the current account with
851
kamaki:
852

    
853
.. code-block:: console
854

    
855
  $ kamaki user authenticate
856

    
857
This will output user information.
858

    
859
Upload Image
860
------------
861

    
862
By convention, images are stored in a container called ``images``. Check if the
863
container exists, by listing all containers in your account:
864

    
865
.. code-block:: console
866

    
867
   $ kamaki file list
868

    
869
If the container ``images`` does not exist, create it:
870

    
871
.. code-block:: console
872

    
873
  $ kamaki file create images
874

    
875
You are now ready to upload an image to container ``images``. You can upload it
876
with a Pithos+ client, or use kamaki directly:
877

    
878
.. code-block:: console
879

    
880
   $ kamaki file upload ubuntu.iso images
881

    
882
You can use any Pithos+ client to verify that the image was uploaded correctly,
883
or you can list the contents of the container with kamaki:
884

    
885
.. code-block:: console
886

    
887
  $ kamaki file list images
888

    
889
The full Pithos URL for the previous example will be
890
``pithos://u53r-un1qu3-1d/images/ubuntu.iso`` where ``u53r-un1qu3-1d`` is the
891
unique user id (uuid).
892

    
893
Register Image
894
--------------
895

    
896
To register an image you will need to use the full Pithos+ URL. To register as
897
a public image the one from the previous example use:
898

    
899
.. code-block:: console
900

    
901
   $ kamaki image register Ubuntu pithos://u53r-un1qu3-1d/images/ubuntu.iso --public
902

    
903
The ``--public`` flag is important, if missing the registered image will not
904
be listed by ``kamaki image list``.
905

    
906
Use ``kamaki image register`` with no arguments to see a list of available
907
options. A more complete example would be the following:
908

    
909
.. code-block:: console
910

    
911
   $ kamaki image register Ubuntu pithos://u53r-un1qu3-1d/images/ubuntu.iso \
912
            --public --disk-format diskdump --property kernel=3.1.2
913

    
914
To verify that the image was registered successfully use:
915

    
916
.. code-block:: console
917

    
918
   $ kamaki image list --name-like=ubuntu
919

    
920

    
921
Miscellaneous
922
=============
923

    
924
.. RabbitMQ
925

    
926
RabbitMQ Broker
927
---------------
928

    
929
Queue nodes run the RabbitMQ sofware, which provides AMQP functionality. To
930
guarantee high-availability, more than one Queue nodes should be deployed, each
931
of them belonging to the same `RabbitMQ cluster
932
<http://www.rabbitmq.com/clustering.html>`_. Synnefo uses the RabbitMQ
933
active/active `High Available Queues <http://www.rabbitmq.com/ha.html>`_ which
934
are mirrored between two nodes within a RabbitMQ cluster.
935

    
936
The RabbitMQ nodes that form the cluster, are declared to Synnefo through the
937
`AMQP_HOSTS` setting. Each time a Synnefo component needs to connect to
938
RabbitMQ, one of these nodes is chosen in a random way. The client that Synnefo
939
uses to connect to RabbitMQ, handles connection failures transparently and
940
tries to reconnect to a different node. As long as one of these nodes are up
941
and running, functionality of Synnefo should not be downgraded by the RabbitMQ
942
node failures.
943

    
944
All the queues that are being used are declared as durable, meaning that
945
messages are persistently stored to RabbitMQ, until they get successfully
946
processed by a client.
947

    
948
Currently, RabbitMQ is used by the following components:
949

    
950
* `snf-ganeti-eventd`, `snf-ganeti-hook` and `snf-progress-monitor`:
951
  These components send messages concerning the status and progress of
952
  jobs in the Ganeti backend.
953
* `snf-dispatcher`: This daemon, consumes the messages that are sent from
954
  the above components, and updates the Cyclades DB accordingly.
955

    
956

    
957
Installation
958
~~~~~~~~~~~~
959

    
960
Please check the RabbitMQ documentation which covers extensively the
961
`installation of RabbitMQ server <http://www.rabbitmq.com/download.html>`_ and
962
the setup of a `RabbitMQ cluster <http://www.rabbitmq.com/clustering.html>`_.
963
Also, check out the `web management plugin
964
<http://www.rabbitmq.com/management.html>`_ that can be useful for managing and
965
monitoring RabbitMQ.
966

    
967
For a basic installation of RabbitMQ on two nodes (node1 and node2) you can do
968
the following:
969

    
970
On both nodes, install rabbitmq-server and create a Synnefo user:
971

    
972
.. code-block:: console
973

    
974
  $ apt-get install rabbitmq-server
975
  $ rabbitmqctl add_user synnefo "example_pass"
976
  $ rabbitmqctl set_permissions synnefo  ".*" ".*" ".*"
977

    
978
Also guarantee that both nodes share the same cookie, by running:
979

    
980
.. code-block:: console
981

    
982
  $ scp node1:/var/lib/rabbitmq/.erlang.cookie node2:/var/lib/rabbitmq/.erlang.cookie
983

    
984
and restart the nodes:
985

    
986
.. code-block:: console
987

    
988
  $ /etc/init.d/rabbitmq-server restart
989

    
990

    
991
To setup the RabbitMQ cluster run:
992

    
993
.. code-block:: console
994

    
995
  root@node2: rabbitmqctl stop_app
996
  root@node2: rabbitmqctl reset
997
  root@node2: rabbitmqctl cluster rabbit@node1 rabbit@node2
998
  root@node2: rabbitmqctl start_app
999

    
1000
You can verify that the cluster is set up correctly by running:
1001

    
1002
.. code-block:: console
1003

    
1004
  root@node2: rabbitmqctl cluster_status
1005

    
1006

    
1007

    
1008

    
1009

    
1010
Admin tool: snf-manage
1011
----------------------
1012

    
1013
``snf-manage`` is a tool used to perform various administrative tasks. It needs
1014
to be able to access the django database, so the following should be able to
1015
import the Django settings.
1016

    
1017
Additionally, administrative tasks can be performed via the admin web interface
1018
located in /admin. Only users of type ADMIN can access the admin pages. To
1019
change the type of a user to ADMIN, snf-manage can be used:
1020

    
1021
.. code-block:: console
1022

    
1023
   $ snf-manage user-modify 42 --type ADMIN
1024

    
1025
Logging
1026
-------
1027

    
1028
Logging in Synnefo is using Python's logging module. The module is configured
1029
using dictionary configuration, whose format is described here:
1030

    
1031
http://docs.python.org/release/2.7.1/library/logging.html#logging-config-dictschema
1032

    
1033
Note that this is a feature of Python 2.7 that we have backported for use in
1034
Python 2.6.
1035

    
1036
The logging configuration dictionary is defined in
1037
``/etc/synnefo/10-snf-webproject-logging.conf``
1038

    
1039
The administrator can have finer logging control by modifying the
1040
``LOGGING_SETUP`` dictionary, and defining subloggers with different handlers
1041
and log levels.  e.g. To enable debug messages only for the API set the level
1042
of 'synnefo.api' to ``DEBUG``
1043

    
1044
By default, the Django webapp and snf-manage logs to syslog, while
1045
`snf-dispatcher` logs to `/var/log/synnefo/dispatcher.log`.
1046

    
1047

    
1048
.. _scale-up:
1049

    
1050
Scaling up to multiple nodes
1051
============================
1052

    
1053
Here we will describe how should a large scale Synnefo deployment look like. Make
1054
sure you are familiar with Synnefo and Ganeti before proceeding with this section.
1055
This means you should at least have already set up successfully a working Synnefo
1056
deployment as described in the :ref:`Admin's Quick Installation Guide
1057
<quick-install-admin-guide>` and also read the Administrator's Guide until this
1058
section.
1059

    
1060
Graph of a scale-out Synnefo deployment
1061
---------------------------------------
1062

    
1063
Each box in the following graph corresponds to a distinct physical node:
1064

    
1065
.. image:: images/synnefo-arch2-roles.png
1066
   :width: 100%
1067
   :target: _images/synnefo-arch2-roles.png
1068

    
1069
The above graph is actually the same with the one at the beginning of this
1070
:ref:`guide <admin-guide>`, with the only difference that here we show the
1071
Synnefo roles of each physical node. These roles are described in the
1072
following section.
1073

    
1074
.. _physical-node-roles:
1075

    
1076
Physical Node roles
1077
-------------------
1078

    
1079
As appears in the previous graph, a scale-out Synnefo deployment consists of
1080
multiple physical nodes that have the following roles:
1081

    
1082
* **WEBSERVER**: A web server running in front of gunicorn (e.g.: Apache, nginx)
1083
* **ASTAKOS**: The Astakos application (gunicorn)
1084
* **ASTAKOS_DB**: The Astakos database (postgresql)
1085
* **PITHOS**: The Pithos application (gunicorn)
1086
* **PITHOS_DB**: The Pithos database (postgresql)
1087
* **CYCLADES**: The Cyclades application (gunicorn)
1088
* **CYCLADES_DB**: The Cyclades database (postgresql)
1089
* **MQ**: The message queue (RabbitMQ)
1090
* **GANETI_MASTER**: The Ganeti master of a Ganeti cluster
1091
* **GANETI_NODE** : A VM-capable Ganeti node of a Ganeti cluster
1092

    
1093
You will probably also have:
1094

    
1095
* **CMS**: The CMS used as a frotend portal for the Synnefo services
1096
* **NS**: A nameserver serving all other Synnefo nodes and resolving Synnefo FQDNs
1097
* **CLIENT**: A machine that runs the Synnefo clients (e.g.: kamaki, Web UI),
1098
              most of the times, the end user's local machine
1099

    
1100
From this point we will also refer to the following groups of roles:
1101

    
1102
* **SYNNEFO**: [ **ASTAKOS**, **ASTAKOS_DB**, **PITHOS**, **PITHOS_DB**, **CYCLADES**, **CYCLADES_DB**, **MQ**, **CMS**]
1103
* **G_BACKEND**: [**GANETI_MASTER**, **GANETI_NODE**]
1104

    
1105
Of course, when deploying Synnefo you can combine multiple of the above roles on a
1106
single physical node, but if you are trying to scale out, the above separation
1107
gives you significant advantages.
1108

    
1109
So, in the next section we will take a look on what components you will have to
1110
install on each physical node depending on its Synnefo role. We assume the graph's
1111
architecture.
1112

    
1113
Components for each role
1114
------------------------
1115

    
1116
When deploying Synnefo in large scale, you need to install different Synnefo
1117
or/and third party components on different physical nodes according to their
1118
Synnefo role, as stated in the previous section.
1119

    
1120
Specifically:
1121

    
1122
Role **WEBSERVER**
1123
    * Synnefo components: `None`
1124
    * 3rd party components: Apache
1125
Role **ASTAKOS**
1126
    * Synnefo components: `snf-webproject`, `snf-astakos-app`
1127
    * 3rd party components: Django, Gunicorn
1128
Role **ASTAKOS_DB**
1129
    * Synnefo components: `None`
1130
    * 3rd party components: PostgreSQL
1131
Role **PITHOS**
1132
    * Synnefo components: `snf-webproject`, `snf-pithos-app`, `snf-pithos-webclient`
1133
    * 3rd party components: Django, Gunicorn
1134
Role **PITHOS_DB**
1135
    * Synnefo components: `None`
1136
    * 3rd party components: PostgreSQL
1137
Role **CYCLADES**
1138
    * Synnefo components: `snf-webproject`, `snf-cyclades-app`, `snf-vncauthproxy`
1139
    * 3rd party components: Django Gunicorn
1140
Role **CYCLADES_DB**
1141
    * Synnefo components: `None`
1142
    * 3rd party components: PostgreSQL
1143
Role **MQ**
1144
    * Synnefo components: `None`
1145
    * 3rd party components: RabbitMQ
1146
Role **GANETI_MASTER**
1147
    * Synnefo components: `snf-cyclades-gtools`
1148
    * 3rd party components: Ganeti
1149
Role **GANETI_NODE**
1150
    * Synnefo components: `snf-cyclades-gtools`, `snf-network`, `snf-image`, `nfdhcpd`
1151
    * 3rd party components: Ganeti
1152
Role **CMS**
1153
    * Synnefo components: `snf-webproject`, `snf-cloudcms`
1154
    * 3rd party components: Django, Gunicorn
1155
Role **NS**
1156
    * Synnefo components: `None`
1157
    * 3rd party components: BIND
1158
Role **CLIENT**
1159
    * Synnefo components: `kamaki`, `snf-image-creator`
1160
    * 3rd party components: `None`
1161

    
1162
Example scale out installation
1163
------------------------------
1164

    
1165
In this section we describe an example of a medium scale installation which
1166
combines multiple roles on 10 different physical nodes. We also provide a
1167
:ref:`guide <i-synnefo>` to help with such an install.
1168

    
1169
We assume that we have the following 10 physical nodes with the corresponding
1170
roles:
1171

    
1172
Node1:
1173
    **WEBSERVER**, **ASTAKOS**
1174
      Guide sections:
1175
        * :ref:`apt <i-apt>`
1176
        * :ref:`gunicorn <i-gunicorn>`
1177
        * :ref:`apache <i-apache>`
1178
        * :ref:`snf-webproject <i-webproject>`
1179
        * :ref:`snf-astakos-app <i-astakos>`
1180
Node2:
1181
    **WEBSERVER**, **PITHOS**
1182
      Guide sections:
1183
        * :ref:`apt <i-apt>`
1184
        * :ref:`gunicorn <i-gunicorn>`
1185
        * :ref:`apache <i-apache>`
1186
        * :ref:`snf-webproject <i-webproject>`
1187
        * :ref:`snf-pithos-app <i-pithos>`
1188
        * :ref:`snf-pithos-webclient <i-pithos>`
1189
Node3:
1190
    **WEBSERVER**, **CYCLADES**
1191
      Guide sections:
1192
        * :ref:`apt <i-apt>`
1193
        * :ref:`gunicorn <i-gunicorn>`
1194
        * :ref:`apache <i-apache>`
1195
        * :ref:`snf-webproject <i-webproject>`
1196
        * :ref:`snf-cyclades-app <i-cyclades>`
1197
        * :ref:`snf-vncauthproxy <i-cyclades>`
1198
Node4:
1199
    **WEBSERVER**, **CMS**
1200
      Guide sections:
1201
        * :ref:`apt <i-apt>`
1202
        * :ref:`gunicorn <i-gunicorn>`
1203
        * :ref:`apache <i-apache>`
1204
        * :ref:`snf-webproject <i-webproject>`
1205
        * :ref:`snf-cloudcms <i-cms>`
1206
Node5:
1207
    **ASTAKOS_DB**, **PITHOS_DB**, **CYCLADES_DB**
1208
      Guide sections:
1209
        * :ref:`apt <i-apt>`
1210
        * :ref:`postgresql <i-db>`
1211
Node6:
1212
    **MQ**
1213
      Guide sections:
1214
        * :ref:`apt <i-apt>`
1215
        * :ref:`rabbitmq <i-mq>`
1216
Node7:
1217
    **GANETI_MASTER**, **GANETI_NODE**
1218
      Guide sections:
1219
        * :ref:`apt <i-apt>`
1220
        * :ref:`general <i-backends>`
1221
        * :ref:`ganeti <i-ganeti>`
1222
        * :ref:`snf-cyclades-gtools <i-gtools>`
1223
        * :ref:`snf-network <i-network>`
1224
        * :ref:`snf-image <i-image>`
1225
        * :ref:`nfdhcpd <i-network>`
1226
Node8:
1227
    **GANETI_NODE**
1228
      Guide sections:
1229
        * :ref:`apt <i-apt>`
1230
        * :ref:`general <i-backends>`
1231
        * :ref:`ganeti <i-ganeti>`
1232
        * :ref:`snf-cyclades-gtools <i-gtools>`
1233
        * :ref:`snf-network <i-network>`
1234
        * :ref:`snf-image <i-image>`
1235
        * :ref:`nfdhcpd <i-network>`
1236
Node9:
1237
    **GANETI_NODE**
1238
      Guide sections:
1239
        `Same as Node8`
1240
Node10:
1241
    **GANETI_NODE**
1242
      Guide sections:
1243
        `Same as Node8`
1244

    
1245
All sections: :ref:`Scale out Guide <i-synnefo>`
1246

    
1247

    
1248
Upgrade Notes
1249
=============
1250

    
1251
.. toctree::
1252
   :maxdepth: 1
1253

    
1254
   v0.12 -> v0.13 <upgrade/upgrade-0.13>
1255

    
1256

    
1257
Changelog, NEWS
1258
===============
1259

    
1260
* v0.13 :ref:`Changelog <Changelog-0.13>`, :ref:`NEWS <NEWS-0.13>`