Synnefo

cyclades: Fix bug in snf-dispatcher logic

This commit reverts commits #07602322 and #30514662 which introduced a
bug in dispatcher's logic. These commit fixed the issue that, in a case
the Cyclades DB is unsynced with Ganeti, a resource that does not exist...

cyclades: Fix corner case in snf-dispatcher

Delete network from DB when job fails but the action of network is
DESTROY. This case appears when someone tries to remove a network that
has been deleted from Ganeti backend, but not yet from DB.

cyclades: Wrap diagnostic creation in transaction

Fix bug in snf-dispatcher causing wrong updates of VMs from ganeti.
Creation if instance diagnostics must run inside a transaction, because
otherwise, the lock of the vm (select_for_update) is released too early....

cyclades: Speed up reconciliation

Speed up reconciliation by using mulitprocessing to get in parallel
the instances from all Ganeti backends.

cyclades: Speed up server reconciliation

Speed up reconciliation, by increasing the time interval to get Ganeti
job status for VMs in building state.

cyclades: Validate MAC prefix in pool-create cmd

cyclades: Rename service token

Rename CYCLADES_ASTAKOS_SERVICE_TOKEN to CYCLADES_SERVICE_TOKEN for
uniformity.

cyclades: Add 'drained' flag to Network

Extend Network model with the 'drained' attribute, to indicate whether a
network should be used in public IP allocation when creating servers.
This option is only useful for public networks.

cyclades: Add --hypervisor option to backend cmds

Add '--hypervisor' option to 'backend-add' and 'backend-modify'
management commands.

Refs #3772

cyclades: Various fixes to tests

cyclades: simplify comparison of NICs

cyclades: Fix deadlock when connecting vms

Fix deadlock caused by 'get_or_create' method, since it acquires an
implicit shared lock. Take an exclusive lock on network to fix this
deadlock.

cyclades: Simplify reconcile-networks checks

Refactor and simplify checks of reconcile-networks to take advantage
of lazy networks.

cyclades: Update 'backend-add' for lazy networks

Do not create private networks when adding a new backend. Since,
"lazy networks" this is no longer needed.

cyclades: refactor code relative to commission

Refactor Cyclades code that handles commissioning with Quotaholder. This
commit removes the 'uses_commission' decorator. Instead it introduces
the 'issue_and_accept_commission' function that is responsible for...

Remove transaction from 'backend-add' command

Do not run 'backend-add' command inside transaction, because if
'snf-dispatcher' is running, it will receive messages that will arrive
from a Ganeti backend that does not exist in the DB and will not be able
to process them.

Use 'python-objpool' instead of 'synnefo.lib.pool'

Use the python-objpool package for pooling resources as object, instead
of the pooling code that existed in snf-common ('synnefo.lib.pool')

Refs #3524

Create public NIC when creating VM

Until now, a server's public NetworkInterface was not created in the DB
until the arrival of the message from Ganeti hooks. However, the IPv4
address of the NIC is reserved before sending the OP_INSTANCE_CREATE
job to Ganeti....

Move common code for mngmt commands to webproject

Move cyclades functions that are useful for all managament commands, like
pprint_table or parse_filters, to snf-webproject.

Merge branch 'release-0.13' into develop

Conflicts:
version

Merge branch 'release-0.13' into develop

Conflicts:
snf-cyclades-app/synnefo/api/management/commands/cyclades-export-quota.py
snf-cyclades-app/synnefo/api/management/commands/network-list.py
snf-cyclades-app/synnefo/api/management/commands/server-inspect.py...

Improve perf for snf-manage cyclades commands

When the snf-manage {server, network}-list commands are invoked with the
--displayname option, batch the UUID-to-displayname requests to astakos,
instead of issuing a request for each (unique) UUID.

Refs #3375

reconciliation: handle missing network from db

Continue reconciliation even if an orphan network is missing from DB.

pep8 convension

Add --uuids option to snf-manage cmds

Fix reconciliation for building VMs

Since commit 482c645, a VM exists in the Cyclades DB before the
corresponding job is enqueued in the Ganeti backend. Therefore, there is
a corner case, where Cyclades can not enqueue the job to Ganeti and
Cyclades crashes before deleting the VM from the DB, which results in a...

Fix pep8 issues

Fix bug in network reconciliation

Fix bug in network reconciliation caused by wrong variable scope.

Rename ARCHIPELAGOS to ARCHIPELAGO

Rename ARCHIPELAGOS_BACKENDS setting to ARCHIPELAGO_BACKEND.

Network field type has been renamed to flavor

Add warning message for backend without networks

Remove unused imports from tests

Refactor tests

Add more tests for db, logic and api

Add more tests for db and logic

Misc minor bug fixes

Fix variable scope bug in process_net_status

Add backend-id option to reconcile-servers

Make reconcile-servers perform reconciliation only for the specified
backend.

Modify backend-modify --drained and --offline args

Make backend-modify drained argument as --drained=True|False instead
of --drained and --no-drained. The same for offline.

Rename reconcile command to reconcile-servers

Rename `snf-manage reconcile` management command to `snf-manage
reconcile-servers`, to be consistent with `snf-manage
reconcile-networks`. Also, treat '--detect-all' option as default
when no other '--detect-*' is specified.

Implement quotas-init and quotas-verify

Implement quotas-init and quotas-verify management commands.
- quotas-init sets quotas of Quotaholder to the state of quotas in
Cyclades DB
- quotas-verify detects incosistencies between quotas in Quotaholder and...

Fix typos

Merge branch 'origin/devel-0.13' into quota

Conflicts:
snf-cyclades-app/synnefo/api/servers.py

Implement reconciliation for pending quotas

Do not automatically update network state

Each modification in a BackendNetwork object, triggers an update
to the corresponding Network object. Until now this was done by
the BackendNetwork save() method. Now it is explicitly done
from the backend pross_network_status.

Use vmapi when creating servers

Use the vmapi when creating servers. Creete instance RAPI call takes the
configuration url instead all the image parametere. The id and format
of the image are still passed, just for readability and debugging.

Also renamed params_url to config-url.

Implement resizing of pools

Extend PoolManager to support extending and shrinking of Pools. Also
add extra option '--size' to pool-modify command.

Make amqp_puka get logger as argument

Add check for clustername when adding backend

Fix typo