Synnefo

cyclades: Process progress of snapshots

Update status of snapshots based on notifications about the progress of
Ganeti OP_INSTANCE_SNAPSHOT job. Extend snf-ganeti-eventd and
snf-dispatcher to create and handle such notifications. Currently, we
store the snapshot_id, in a hacky way as a reason in the...

cyclades: volume-modify and various fixes

cyclades: Make snf-dispatcher handle volumes

Update snf-dispatcher to handle messages from snf-ganeti-eventd about
the instance disks(volumes). Handling of disks is done almost exactly as
with NICs.

cyclades: Various fixes relative for volumes

• Do not store source_{volume_id, image_id, snapshot_id} as seperate
  fields. Instead store them in the 'source' field, with a special prefix
  to indicate the source type.
• Add 'origin' field to Volume model to contain the origin that will be...

cyclades: Delete volumes together with the instance

cyclades: Implementat volumes and snapshots

cyclades: Fix quota bug in snf-dispatcher

Clear the 'vm.serial' field after processing a job that was not
expected. When processing such a job, we reject the pending commission
and issue a new one that reflects the unexpected job. However, since we
reject the commission that is associated with the 'vm.task_job_id' job,...

cyclades: Fixes causing unsynced quotas

• clear QuotaHolderSerial objects when resolving pending commissions.
• Fix wrong name of Ganeti opcodes.
• Do not update vm's flavor before issuing new commission

cyclades: Optimize dispatcher DB queries

cyclades: Pass arbitrary params to ext providers

Add 'GANETI_DISK_PROVIDER_KWARGS' setting, which contains a dictionary
of arbitrary parameters for each of Ganeti's ExtStorage providers and
which will be passed to the 'disks' option of CreateInstance Ganeti job.

cyclades: Fix wrong state of deleted networks

Before deleting a network, the network is marked as drained to be
excluded from automatic IP allocation. The state of the network that is
exposed in the API is always 'SNF:DRAINED' if the drained flag is set....

Merge branch 'master' into develop

Conflicts:
ci/schemas/one_node_wheezy/wheezy.conf
docs/admin-guide.rst
snf-cyclades-app/synnefo/api/actions.py
snf-cyclades-app/synnefo/api/servers.py
snf-cyclades-app/synnefo/logic/backend.py
snf-cyclades-app/synnefo/logic/callbacks.py...

cyclades: Map reboot type to shutdown timeout

Fix handling of reboot type option. If 'hard' reboot type is specified
this is mapped to zero shutdown timeout of OP_INSTANCE_REBOOT Ganeti
job. This is only supported from Ganeti>=2.10 and snf-ganeti. In other...

cyclades: Add name to Ganeti NICs

Add 'name' attribute to newly created NICs. Currently, the name of the
NICs will not be used to refer to NICs, as it will be in Synnefo 0.15.
Names are added only to ease the migration to version 0.15. Also, this
commit updates snf-dispatcher to preserve the name of the NIC, if any,...

cyclades: Generalize issue_and_accept_commission

Use issue_and_accept_commission for any action where one needs to
automatically accept the commission after issuing and committing.

Merge branch 'hotfix-0.14.10' into develop

Conflicts:
Changelog
snf-cyclades-app/synnefo/api/servers.py
snf-cyclades-app/synnefo/logic/backend.py
snf-cyclades-app/synnefo/logic/callbacks.py
snf-cyclades-app/synnefo/logic/management/commands/reconcile-networks.py...

Merge branch 'feature-cyclades-networks' into develop

Conflicts:
snf-cyclades-app/synnefo/api/management/commands/server-inspect.py
snf-cyclades-app/synnefo/app_settings/urls.py
snf-cyclades-app/synnefo/logic/backend.py
snf-cyclades-app/synnefo/logic/backend_allocator.py...

cyclades: Fix case where 'dfree' is None

If Ganeti has not enabled 'plain' or 'drbd' disk template, then it
reports 'dfree' as None. Fix this corner case by treating None as 0.

cyclades: Remove hotplug from InstanceCreate

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>

cyclades: Remove deprecated network type in ganeti

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>

Merge branch 'master' into develop

Conflicts:
Changelog
astakosclient/astakosclient/keypath.py
ci/utils.py
docs/admin-guide.rst
snf-astakos-app/astakos/im/cookie.py
snf-astakos-app/astakos/im/functions.py
snf-astakos-app/astakos/im/quotas.py
snf-astakos-app/astakos/im/views/projects.py...

cyclades: Pass 'nfdhcp' tag to IPv6 networks

Pass 'nfdhcp' tag even if network is IPv6 only.

cyclades: Create IPv6 Addresses from dispatcher

When ports are created from the API they do not have an IPv6 address.
The IPv6 address is only available after the MAC address of the
NIC is known which is only available after the message from the
Ganeti backend has been received. However, dispatcher did not created...

cyclades: Refactor code

Move code relative with allocating ips and floating IPs to 'logic.ips'
module.

cyclades: Series of fixes regarding networks & ips

• Do not delete the network from the DB if it has associated floating
  IPs. This should never be the case because a check is made before
  deleting the network, however this check is added for safety.
• Do not allow deleting a network if there are still ports configured on...

cyclades: Create ports without 'device_id'

Allow creation of ports that are not connected to any instance. Such
ports will be in 'DOWN' state. To allow this, 'machine' attribute of
NetworkInterface model has been made to allow null values. Finally, the
'logic.ports' module has been remove, and the corresponding code has...

cyclades: Update state of NICs if VM creation fail

Set the state of NICs(ports) to ERROR if the OP_INSTANCE_CREATE job
fails.

cyclades: Create network in backend if missing

Automatically create a network in a Ganeti backend if missing.
Currently, this was done only for private networks. This commit changes
this to be done for any kind of networks.

cyclades: Fix bug when deleting instances

If OP_INSTANCE_REMOVE runs before OP_INSTANCE_CREATE the job will fail.
Dispatcher will ask Ganeti if the instance exists, and if not it will
release it's IPv4 address. However the instance may not exist because it...

cyclades: Refactor backend code for new networks

• Optimize DB queries by using prefetch related for NIC IPs.
• Do not try to construct new NIC if it does not exist in Cyclades DB.
• Update tests

cyclades: Make logic tests pass

Refactor Cyclades code so that it follows the new models for Networks,
Subnets and IPAddresses. This refactor is not complete. Instead is the
necessary refactor for making tests for server API calls succeed.

cyclades: Replace FloatingIP with IPAddress

This commit just replaces all references to FloatingIP objects with
references to IPAddress object, in order to be able to just load the
code. But the code is left unchanged, so all tests will fail.

cyclades: Clear FloatingIP when deleting NIC

Override 'delete' method of NetworkInterface model, to clear the
'machine' flag of the corresponding NIC, if any.

cyclades: Use NIC UUID in firewall tags

Use NICs UUID instead of index to generate firewall tags. Make the
'set_firewall_profile' action to take the NICs ID as argument. Also,
update the firewall settings.

cyclades: Use UUIDs to update VM NICs

Callback functions of snf-dispatcher updated the NICs of a VM by
deleting all NICs from DB, and creating the new NICs that were described
in the message. However, since NICs have UUIDs, this is no longer
required. This commit update callbacks to update the NICs based on their...

cyclades: Fix getting action from Ganeti opcode

Parse the fields of the Ganeti command, to distinquish between
'CONNECT' and 'DISCONNECT' in case of 'OP_INSTANCE_SET_PARAMS'.

cyclades: Use NIC uuid when deleting NIC

Use opportunistic locking

Use opportunistic locking, introduced in Ganeti 2.7, to allow parallel creation
of VMs in the Ganeti backend. When this option is set, Ganeti iallocator
will allocate the VM only to nodes that not locked (e.g. by other VM
creation). This may result in unbalanced clusters, but `hbal` can be...

Pass network type info in tags during create

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>

New style format for modifying a NIC

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>

cyclades: Make disk_templates work with Ganeti<2.8

'enabled_disk_templates' cluster attribute has been added in Ganeti 2.8.
If this attribute is missing, return the ipolicy disk templates.

cyclades: Fix wrong quotas when creating server

Catch exception when talking to Ganeti inside 'server_command',
otherwise the pending commission will be rejected. Also, accept the
commission in case of build, because the VM will be created in DB.

cyclades: refactor server creation command

Split the server creation command in more steps in order to be
decorated with the 'server_command' function and follow the logic of all
other server commands. The main change is that the commission job is not
accepted when the VM is stored in DB, but when the OP_INSTANCE_CREATE...

cyclades: Simplify code

Simplify code by removing unnecessary temp variable.

cyclades: Do not save stale objects after a commit

Call to issue_and_accept_commission saves the resource object (eg a vm)
and commits the transaction. This has the side-effect that the resource
object becomes stale: its lock is released and another thread may acquire...

cyclades: Create NIC before send job to Ganeti

Create NICs in building state before enqueuing corresponding jobs to
Ganeti. This has already been done when creating the instance, but not
when connecting it to other networks.

cyclades: Fix bug in handling of reboot type.

'type' attribute of reboot OS API call was directly mapped to the
'reboot_type' option of Ganeti reboot job. However these two are
semantically different. The 'SOFT' and 'HARD' reboot types of OS API are
mapped to 'shutdown_timeout' option of Ganeti jobs. However, setting...

cyclades: Fix check for firewall profile

cyclades: Add 'name' to all commissions.

Make all quotaholder commissions to have a name. Since, Cyclades does
not store the association of a commission with a resource, we include
the resource in the commission name.

cyclades-rapi: Add bulk option to GetJobs