Synnefo

Perform allocations in separate transactions

Add ForUpdateManager to the Backend model, in order to guarantee
exclusive order when allocating a VM.

Use manual transactions when allocating Backend and public IP for a new
VM, since these allocations need exclusive access (SELECT..FOR UPDATE)...

Merge branch 'public_networks' into devel-0.11

Conflicts:
snf-cyclades-app/synnefo/logic/dispatcher.py

Make snf-dispatcher work with gevent

Remove workers from snf-dispatcher in order to make it compatible with gevent
and specifically the green, monkey patched version of psycopg2
(synnefo.lib.db.psyco_gevent). Also after daemonizing, gevent must be
reinitialized because the deamon context forks.

Change process title to simplify 'ps' display

Use python-setproctitle to change the process title of the running
daemon. Make it appear as a native executable to simplify
administration. Only set executable name, because cannot set individual
values for arguments.

Fix two bugs in network reconciliation

Fix double logging of dispatcher

Dispatcher had as default handlers the console and the file handler.
However, when daemonizing, we redirect the stdout and stderr to the file
handler, which resulted in the log messages to be printed twice. So we
remove the console handler from the dict configuration, and we only add...

Catch corner case in reconciliation

Catch error produced when the DB entry for an orphan VM in Ganeti does
not exist. This may happen because either an entry from DB was removed
or because a VM in Ganeti with the synnefo prefix was created.

Add some logging to backend allocator

Refactor IP pools

Treat IP pool the same way as bridge and mac-prefix pools. The pool is
removed from the Network table and moved to an IPPoolTable containing the
IP pools for all networks. Also added support for external reservations.

Migration files for transfering the pool to the new tables are included....

Management commands for pools

Create three new management commands for listing, showing and modifying
a pool.

Refactor bridge and mac-prefix pool tables

Redesign the implementation of pools and map mac-prefix and bridge pools
to that design.

Each pool type is mapped to a table in the database, and each row of
that table is a pool of that type. All pool tables must contain the...

Enable hotplug during create_instance

This passes hotplug = True to ganeti and enables hotplug per instance.
It checks GANETI_USE_HOTPLUG setting.

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>

Remove dead code

Reserve gateway's IP when creating an IP pool

Add setting indicating whether to use hotplug

Add command for checking pool consistency

Add new management command 'reconcile-pools' which checks the consistency of the
pools and the uniqueness of the associated resources. Currently this
commands detects the inconcistencies, without trying to resolve them.

Implement MAC Prefix Management

In order to guarantee uniqueness of MAC address accross all backends,
synnefo assignes a unique mac prefix to each network in each backend.

A mac-prefix of 20 bits, which is picked from the MacPrefixPool, is
assigned to each Network. The mac-prefix of the network in each...

Get public network by public attribute

Do not find the public network by the primary key, as the initial_data.json
is obsolete. Instead, ise the public attribute of the Network Model.
Currently only one public Network is supported, and the used can not
create or delete the public Network. But this can easily change in the...

Fix bug in process_net_status

Check that nic has an ipv4 before trying to release it.

Consider cpu ratio when allocating vm's

Transaction on create_instance

Implement model manager using row-level db locking

Implement ForUpdateManager that uses SELECT .. FOR UPDATE statement in
order to acquire row-level exclusive locks to DB. Essentially this
manager backports the select_for_update() method that is implemented in...

Display IP pool map in network-inspect command

Add support for IPv6

- Extend the Network model with fields for holding the IPv6 subnet and
gateway.
- Update network API methods.
- Modify ganeti hooks to return IPv6 address for all network interfaces.
IPv6 is now derived from the IPv6 subnet of the network in Ganeti,...

Implement IP management

Delegate IP management to synnefo, in order to be able to allocate IP
addresses accross multiple Ganeti backends. This is achieved by creating a
Pool of IPs for each Network, in the form of a bitarray, which is stored
in the DB as a base64 string....

Rename management commands

Revert "Fix bug in dispatcher callback"

This reverts commit 5d40a128fb593ed641400c462f0e00a1b457f76e.

Implement network reconciliation

Common functions for instance/network backend name

Fix bug about unknown network in instance's nics

Bug fixes concerning networks

Add backend method for removing a vm's nic

Synchronize networks when adding a new backend

Implement BackendNetwork model

Create a new model, to represent the state of each network in each
backend.

Implement backend allocator and management command

New module implementing instance allocation to one of the available
backends. Allocation is performed based on memory and disk utilization.
Statistics for each backend are stored in DB and updated periodically....

Use the Backend client instead of global RAPI

Replace use of global RAPI client, with the cached client from the
VirtualMachine or Backend model.

Dispatch network actions to all backends.

Rename backend_id to backend_vm_id

Rename backend_id field of VirtualMachine model to avoid conflict between the
backend that hosts the VM and the id that the VM will have on the backend.

Rename management command

Simple per user quota mechanism for networks/vms

- New dict setting VMS_USER_QUOTA to allow setting per user quota
(overrides MAX_VMS_PER_USER setting)
- Private networks quota mechanism (based on MAX_NETWORKS_PER_USER and
NETWORKS_USER_QUOTA settings)...

Remove messages that can not be handled from queue

Temprory fix for removing accumulated messages from RabbitMQ. Currently
snf-ganeti-eventd produces messages that can not be handled from
snf-dispatcher (e.g messages not concering an instance). These messages...

Refactor common code in dispatcher's callbacks

Remove unused code

Remove old code used for reconciliation, as the reconciliation mechanism has
changed

Small refactoring in getting vm id

Update modules interacting with AMQP

Update all modules which are interacting with the RabbitMQ messaging system to
use the AMQPClient in order to use highly available queues in active/active
mode.

Extend messages to include a unique identifier in order to handle...

Take advantage of ganeti hotplug

No race condition. No need for ther NetworkInterface to have an
operstate. Hook runs during instance-modify and depicts the currently
existing nics. No reboot needed modification to take place.

Later in mutliple backend mac prefix handling should be done by synnefo....

Simplify disconnecting vm from network

Use the new option of Ganeti, to remove a specific nic.

Remove --detect-orphan-{servers,networks}

They have to depend on Astakos, move their functionality externally.

Add --detect-orphan-{servers,networks}

Remove email queues from dispatcher

Remove obsolete files that were restored due to merge