Synnefo

dispatcher: Handle NICs in callbacks.

Update 'update_db' callback to handle instance NICs together with
instance status.

Also update the relevant tests.

cyclades: Fix deadlock when connecting vms

Fix deadlock caused by 'get_or_create' method, since it acquires an
implicit shared lock. Take an exclusive lock on network to fix this
deadlock.

cyclades: Implement lazy networks

Implement "lazy networks", which are not created to all Ganeti backends
upon creation, but only when an instance that resides in a Ganeti is
connected to that network.

Update 'create_network' to create the Network object in DB without...

Create public NIC when creating VM

Until now, a server's public NetworkInterface was not created in the DB
until the arrival of the message from Ganeti hooks. However, the IPv4
address of the NIC is reserved before sending the OP_INSTANCE_CREATE
job to Ganeti....

Totally remove password from create_instance

Never pass password as an 'osparam' to Ganeti. Always use the vmapi,
even for VMs created with server-create management command.

Merge branch 'master' into devel-0.13

Conflicts:
snf-cyclades-app/synnefo/db/models.py
snf-cyclades-app/synnefo/logic/management/commands/backend-modify.py

Misc minor bug fixes

Add backend-id option to reconcile-servers

Make reconcile-servers perform reconciliation only for the specified
backend.

Fix typos

Merge branch 'origin/devel-0.13' into quota

Conflicts:
snf-cyclades-app/synnefo/api/servers.py

Remove transaction from update_network_state

Remove transaction decorator from update_network_state function, since
this function is only called from process_network_status, which already runs
inside a transaction.

Remove IGNORE_FLAVOR_DISK_SIZES setting

snf-dispatcher: Update NICs only when needed

Do not remove and recreate the NICs of a VM when NICs have not changed.

Process OP_NETWORK_SET_PARAMS Ganeti jobs

Modify snf-ganeti-eventd and snf-dispatcher to handle jobs with
OP_NETWORK_SET_PARAMS opcode. Handling of these jobs is needed
to update the IP Pool of network, when reserving and releasing
IPs directly from the Ganeti backend.

Do not allow any API action on suspended VMs

Do not allow any modification of any administratively suspended VM through the
API.

Merge branch 'devel-0.12' into ui-0.11

Do not give IPs to networks without dhcp

Refactor process op and network status callbacks

Release address only if a NIC has an IP

Add missing ipv6 arguments to create_network

Hide image password from log file

Pool Ganeti RAPI clients

- Move Ganeti RAPI client from snf-common to snf-cyclades-app/logic.
- Create GanetiRapiClientPool for pooling GanetiRapiClient objects
- Create context manager in db.models for pool handling
- Update code using the RAPI client

Perform allocations in separate transactions

Add ForUpdateManager to the Backend model, in order to guarantee
exclusive order when allocating a VM.

Use manual transactions when allocating Backend and public IP for a new
VM, since these allocations need exclusive access (SELECT..FOR UPDATE)...

Refactor bridge and mac-prefix pool tables

Redesign the implementation of pools and map mac-prefix and bridge pools
to that design.

Each pool type is mapped to a table in the database, and each row of
that table is a pool of that type. All pool tables must contain the...

Implement multiple public networks

Implement multiple public networks in order to support:
- different public networks per backend
- non-consecutive IP ranges in each backend

To support this, public networks are treated different from private
ones. Namely, they are correlated with one backend (and so have...

Logic backend api helper to create vm diagnostics

Support for ext disk providers

Extract provider from flavor disk template name and set the additional
`provider` parameter in disk options. Flavor disk_template should be
formated as `ext_<provider_name>`.

Implement MAC Prefix Management

In order to guarantee uniqueness of MAC address accross all backends,
synnefo assignes a unique mac prefix to each network in each backend.

A mac-prefix of 20 bits, which is picked from the MacPrefixPool, is
assigned to each Network. The mac-prefix of the network in each...

Fix creation of networks in new backend

Setting for using IP Pool to public network

Introduce new setting 'PUBLIC_ROUTED_USE_POOL', to indicate whether
synnefo will be responsible for the IP management of the public network,
or this responsibility will be delegated to Ganeti.

Implement model manager using row-level db locking

Implement ForUpdateManager that uses SELECT .. FOR UPDATE statement in
order to acquire row-level exclusive locks to DB. Essentially this
manager backports the select_for_update() method that is implemented in...

Release IP addresses when nics are destroyed

Implement IP management

Delegate IP management to synnefo, in order to be able to allocate IP
addresses accross multiple Ganeti backends. This is achieved by creating a
Pool of IPs for each Network, in the form of a bitarray, which is stored
in the DB as a base64 string....

Add dirty bit to NetworkInterface

Extend NetworkInterface model with a dirty bit, indicating whether
the index of the NIC is consistent with the one in Ganeti. This flag is
set when deleting a NIC, and is switched off after processing the
notification from Ganeti. Removing a NIC with the dirty bit on is not...

Remove machine if ganeti job fails

Set the VM deleted flag when receiving a failure notification for a
Ganeti OP_INSTANCE_REMOVE operation, even if the VM operstate is not
ERROR. The server may not exist in the backend not only because the
creation failed, but also because a remove notification was lost.

Add backend method for removing a vm's nic

Synchronize networks when adding a new backend

Implement backend allocator and management command

New module implementing instance allocation to one of the available
backends. Allocation is performed based on memory and disk utilization.
Statistics for each backend are stored in DB and updated periodically....

New management commands concerning backends

- Create command to create a backend (snf-manage backend_create)
- Create command to modify a backend (snf-manage backend_list)
- Create command to list backends (snf-manage backend_modify)
- Modify listservers command to display the backend that hosts the VM

Merge branch 'simplequotas' into queues+minor_net

Update modules interacting with AMQP

Update all modules which are interacting with the RabbitMQ messaging system to
use the AMQPClient in order to use highly available queues in active/active
mode.

Extend messages to include a unique identifier in order to handle...

Take advantage of ganeti hotplug

No race condition. No need for ther NetworkInterface to have an
operstate. Hook runs during instance-modify and depicts the currently
existing nics. No reboot needed modification to take place.

Later in mutliple backend mac prefix handling should be done by synnefo....

Simplify disconnecting vm from network

Use the new option of Ganeti, to remove a specific nic.

Switch to the logging middleware

Renamed snf-app to snf-cyclades-app