Statistics
| Branch: | Tag: | Revision:

root / snf-cyclades-app / synnefo / logic / backend.py @ ece5581b

History | View | Annotate | Download (31.9 kB)

# Date Author Comment
f8675683 07/02/2013 12:31 pm Christos Stavrakakis

cyclades: Various fixes regarding floating IPs

Fix some issues regarding floating ips introduced by rebasing branch.

cb66110b 07/02/2013 12:31 pm Christos Stavrakakis

cyclades: Add network policy at server create

Add 'DEFAULT_INSTANCE_NETWORKS' setting that can be used by the
administrator to define the list of networks that each server that is
created must be connected to. Each created VM will have a NIC connected
to each of these networks. This settings can include a list of network...

72dea98f 07/02/2013 12:31 pm Christos Stavrakakis

cyclades: Do not release floating IPs

Update 'release_instance_nics' backend function which is called to
process NIC modifications in the Ganeti backend, to not return a
IPv4 address back to pool if this address is a floating IP.

2c022086 07/02/2013 12:31 pm Christos Stavrakakis

cyclades: Support many NICs when creating vms

Support arbitraty number of NICs when creating Ganeti instances.

41a7fae7 07/02/2013 12:31 pm Christos Stavrakakis

cyclades: Major refactor to api and logic apps

Major refactor to cyclades api and logic apps that removes most code
from the api views and moves it to logic app, so that the same logic
code can be used by different apps (e.g. EC2 API). Code is moved into...

2cd3f389 07/02/2013 12:31 pm Christos Stavrakakis

cyclades: Add backend function for resizing vm

Add 'resize_instance' backend function for resizing a vm. This command
issues an OP_INSTANCE_MODIFY job with the specified beparams, which
currently are vcpus, minmem and maxmem.

Refs #3870

ca4d59e3 07/02/2013 12:31 pm Christos Stavrakakis

cyclades: Handle Ganeti msgs for resized VMs

Handle 'OP_INSTANCE_SET_PARAMS' notifications from Ganeti that have
'beparams' attribute, which corresponds to resized VMs. In such
case, the corresponding flavor is found an the VM in DB is updated to
have this flavor....

e77a29ab 06/28/2013 08:45 pm Christos Stavrakakis

cyclades: Fixes relative to reconciliation

Get available backends once and work only with them, since available
backends may change during reconciliation. Also temporary remove support
for communicating with each backend in a separate process.

79f65825 06/21/2013 01:40 pm Christos Stavrakakis

cyclades: Handle errors when using multiprocessing

Check the process exitcode to determine if a process finished
successfully.

d986cb32 06/21/2013 12:54 pm Christos Stavrakakis

cyclades: Speed up reconciliation

Speed up reconciliation by using mulitprocessing to get in parallel
the instances from all Ganeti backends.

40d53b77 06/19/2013 11:07 am Christos Stavrakakis

cyclades: Fix deadlock when creating servers

Fix deadlock between snf-dispatcher and backend allocator, by making
snf-dispatcher take an exclusive lock to the backend which hosts the vm,
before trying to get the exclusive locks to the IP pools in order to...

b2222a7f 06/13/2013 05:10 pm Christos Stavrakakis

cyclades: Pass disk origin for all 'ext' flavors

bd87213f 05/21/2013 02:07 pm Christos Stavrakakis

cyclades: Sypport backends with Xen hypervisor

Support Ganeti backends with Xen hypervisor:
  • Extend 'Backend' model with the 'hypervisor' attribute, which
    currently can be 'kvm' or 'xen'. Use 'kvm' as default value to
    easy database migration. Also add the corresponding schema migration....
fd95834e 05/14/2013 01:25 pm Christos Stavrakakis

dispatcher: Handle NICs in callbacks.

Update 'update_db' callback to handle instance NICs together with
instance status.

Also update the relevant tests.

be4d8aed 05/14/2013 01:25 pm Christos Stavrakakis

cyclades: simplify comparison of NICs

acfc71ef 05/14/2013 01:25 pm Christos Stavrakakis

cyclades: Fix deadlock when connecting vms

Fix deadlock caused by 'get_or_create' method, since it acquires an
implicit shared lock. Take an exclusive lock on network to fix this
deadlock.

27cda06b 05/14/2013 01:24 pm Christos Stavrakakis

cyclades: Fixes relevant to lazy networks

99af08a4 05/13/2013 12:38 pm Christos Stavrakakis

cyclades: Implement lazy networks

Implement "lazy networks", which are not created to all Ganeti backends
upon creation, but only when an instance that resides in a Ganeti is
connected to that network.

Update 'create_network' to create the Network object in DB without...

2509ce17 04/30/2013 05:05 pm Christos Stavrakakis

cyclades: refactor code relative to commission

Refactor Cyclades code that handles commissioning with Quotaholder. This
commit removes the 'uses_commission' decorator. Instead it introduces
the 'issue_and_accept_commission' function that is responsible for...

939d71dd 03/21/2013 05:05 pm Christos Stavrakakis

Create public NIC when creating VM

Until now, a server's public NetworkInterface was not created in the DB
until the arrival of the message from Ganeti hooks. However, the IPv4
address of the NIC is reserved before sending the OP_INSTANCE_CREATE
job to Ganeti....

e18c1749 03/08/2013 04:22 pm Christos Stavrakakis

Bypass commissioning for orphan networks

Cyclades public networks do not have an owner, and since can not use
the commissioning system. Also, make snf-dispatcher instead of failing
for networks without userid to just a log a warning. This does not apply for...

8528b8ac 02/14/2013 11:03 am Christos Stavrakakis

Totally remove password from create_instance

Never pass password as an 'osparam' to Ganeti. Always use the vmapi,
even for VMs created with server-create management command.

cc92b70f 02/06/2013 02:18 pm Christos Stavrakakis

Fix pep8 warnings

3f77e63d 01/07/2013 12:22 pm Christos Stavrakakis

Merge branch 'master' into devel-0.13

Conflicts:
snf-cyclades-app/synnefo/db/models.py
snf-cyclades-app/synnefo/logic/management/commands/backend-modify.py

5480daec 12/20/2012 05:22 pm Christos Stavrakakis

Remove dummy function

e6f6627c 12/14/2012 06:22 pm Christos Stavrakakis

Misc minor bug fixes

f45a7ac4 12/12/2012 02:05 pm Christos Stavrakakis

Fix variable scope bug in process_net_status

c414bc87 12/10/2012 04:14 pm Christos Stavrakakis

Add backend-id option to reconcile-servers

Make reconcile-servers perform reconciliation only for the specified
backend.

b7d38981 12/04/2012 04:51 pm Dimitris Aragiorgis

Introduce network flavors to replace network types

Add tags and mode to network db model. Replace existing type with
flavor. Introduce 4 different flavors. Each flavor has mode, link,
mac_prefix, tags. Supported flavors are CUSTOM, IP_LESS_ROUTED,
MAC_FILTERED, PHYSICAL_VLAN and are hardcoded in db models....

006c6249 11/30/2012 12:49 pm Christos Stavrakakis

Fix typos

2a599282 11/30/2012 12:20 pm Christos Stavrakakis

Merge handling of ext_<provider> disk templates

33050118 11/29/2012 03:55 pm Georgios D. Tsoukalas

Merge branch 'origin/devel-0.13' into quota

Conflicts:
snf-cyclades-app/synnefo/api/servers.py

cb4eee84 11/26/2012 12:40 pm Christos Stavrakakis

Issue commission when deleting VM/Network

Issue commission to the Quotaholder when being notified from the Ganeti
backend that a VM/Network has been deleted. Although the resource has
already been deleting, we are interacting with the quotaholder in a
two step phase: issue commission and accept commission. It would be...

10de1102 11/26/2012 12:40 pm Christos Stavrakakis

Remove transaction from update_network_state

Remove transaction decorator from update_network_state function, since
this function is only called from process_network_status, which already runs
inside a transaction.

4d5d0b9c 11/26/2012 12:31 pm Christos Stavrakakis

Do not automatically update network state

Each modification in a BackendNetwork object, triggers an update
to the corresponding Network object. Until now this was done by
the BackendNetwork save() method. Now it is explicitly done
from the backend pross_network_status.

d7841399 11/23/2012 03:13 pm Christos Stavrakakis

Remove IGNORE_FLAVOR_DISK_SIZES setting

79b7dbb7 11/23/2012 01:50 pm Christos Stavrakakis

Use vmapi when creating servers

Use the vmapi when creating servers. Creete instance RAPI call takes the
configuration url instead all the image parametere. The id and format
of the image are still passed, just for readability and debugging.

Also renamed params_url to config-url.

b578d9e7 11/22/2012 01:26 pm Christos Stavrakakis

snf-dispatcher: Update NICs only when needed

Do not remove and recreate the NICs of a VM when NICs have not changed.

7f2dbcad 11/07/2012 12:52 pm Christos Stavrakakis

Use select_for_update for VirtualMachines

Use select_for_update method when accessing VM objects to prevent
race conditions. Also move start_action from logic to api.

fd2bdbb2 11/07/2012 11:04 am Christos Stavrakakis

Process OP_NETWORK_SET_PARAMS Ganeti jobs

Modify snf-ganeti-eventd and snf-dispatcher to handle jobs with
OP_NETWORK_SET_PARAMS opcode. Handling of these jobs is needed
to update the IP Pool of network, when reserving and releasing
IPs directly from the Ganeti backend.

2d762302 11/06/2012 12:57 pm Dimitris Aragiorgis

Minor backend fixes in networks

Add conflicts_check option when creating and add network. Check only
if a network is public.

Let tags argument in rapi calls be a list.

Signed-off-by: Dimitris Aragiorgis <>

e221ade2 11/01/2012 01:24 pm Christos Stavrakakis

Do not allow any API action on suspended VMs

Do not allow any modification of any administratively suspended VM through the
API.

1e2cc762 10/29/2012 02:28 pm Kostas Papadimitriou

Merge remote-tracking branch 'origin/devel-0.12' into ui-0.11

eeffcbea 10/23/2012 04:28 pm Kostas Papadimitriou

Merge branch 'devel-0.12' into ui-0.11

40ef487d 10/23/2012 03:50 pm Christos Stavrakakis

Fix bug, UI not updating NICs

Dummy-save the Network when processing messages from Ganeti hooks
about a VM NICs. This is needed, because the UI uses the HTTP changes-sinces
when requesting VMs and Networks, and combines the result in order to show
the new NICs. Not updating the Network object, resulted in VMs IPv4...

87920bc3 10/16/2012 08:50 pm Christos Stavrakakis

Do not give IPs to networks without dhcp

122c4019 10/15/2012 12:56 pm Christos Stavrakakis

Fix bug in reconcile-networks

Fix bug when connecting a network to a nodegroup in network
reconciliation. Use connect_network instead of obsolete
connect_network_group method. Also fix job dependency handling in this
method.

e97288bc 10/15/2012 12:55 pm Christos Stavrakakis

Refactor process op and network status callbacks

e60b4630 10/10/2012 01:36 pm Christos Stavrakakis

Fix bug when synchronously creating networks

fa454545 10/10/2012 01:05 pm Christos Stavrakakis

Release address only if a NIC has an IP

b16db49b 10/09/2012 04:27 pm Kostas Papadimitriou

Merge branch 'devel-0.11' into ui-0.11

Conflicts:
snf-cyclades-app/synnefo/db/models.py
snf-cyclades-app/synnefo/logic/backend.py
snf-cyclades-app/synnefo/ui/static/snf/js/models.js

6cc3a31c 10/04/2012 03:23 pm Christos Stavrakakis

Add missing ipv6 arguments to create_network

c4ce868e 10/04/2012 01:17 pm Christos Stavrakakis

Fix bug in snf-dispatcher about ignored messages

Update backendtime only for jobs that have been successfully completed,
since only those jobs update the state of the VM. Else a "race
condition" may occur when a successful job (e.g. OP_INSTANCE_REMOVE)
completes before an error job for the same VM and messages arrive in...

6afeb85d 10/03/2012 03:58 pm Christos Stavrakakis

Hide image password from log file

bf5c82dc 10/03/2012 01:09 pm Christos Stavrakakis

Add extra logging

3524241a 10/03/2012 12:22 pm Christos Stavrakakis

Pool Ganeti RAPI clients

- Move Ganeti RAPI client from snf-common to snf-cyclades-app/logic.
- Create GanetiRapiClientPool for pooling GanetiRapiClient objects
- Create context manager in db.models for pool handling
- Update code using the RAPI client

27d6d48d 10/01/2012 11:13 am Stratos Psomadakis

Add support for Archipelagos/extstorage COW images

adc46059 09/28/2012 12:19 pm Christos Stavrakakis

Perform allocations in separate transactions

Add ForUpdateManager to the Backend model, in order to guarantee
exclusive order when allocating a VM.

Use manual transactions when allocating Backend and public IP for a new
VM, since these allocations need exclusive access (SELECT..FOR UPDATE)...

fdc94944 09/20/2012 12:14 pm Christos Stavrakakis

Refactor IP pools

Treat IP pool the same way as bridge and mac-prefix pools. The pool is
removed from the Network table and moved to an IPPoolTable containing the
IP pools for all networks. Also added support for external reservations.

Migration files for transfering the pool to the new tables are included....

03992c72 09/19/2012 01:16 pm Christos Stavrakakis

Refactor bridge and mac-prefix pool tables

Redesign the implementation of pools and map mac-prefix and bridge pools
to that design.

Each pool type is mapped to a table in the database, and each row of
that table is a pool of that type. All pool tables must contain the...

341c818e 09/07/2012 05:41 pm Dimitris Aragiorgis

Enable hotplug during create_instance

This passes hotplug = True to ganeti and enables hotplug per instance.
It checks GANETI_USE_HOTPLUG setting.

Signed-off-by: Dimitris Aragiorgis <>

7fede91e 09/07/2012 05:36 pm Christos Stavrakakis

Implement multiple public networks

Implement multiple public networks in order to support:
- different public networks per backend
- non-consecutive IP ranges in each backend

To support this, public networks are treated different from private
ones. Namely, they are correlated with one backend (and so have...

f8d1eaf4 08/09/2012 05:00 pm Kostas Papadimitriou

Revert commented line in logic backend

the line was uncommented by mistake in `7d43565`

7d43565f 08/08/2012 08:27 pm Kostas Papadimitriou

Logic backend api helper to create vm diagnostics

0827883e 08/06/2012 06:17 pm Nikos Skalkotos

Comply with the new monitoring system of snf-image

For now only the image-copy-progress messages are processed

296682fe 08/06/2012 11:45 am Kostas Papadimitriou

Support for ext disk providers

Extract provider from flavor disk template name and set the additional
`provider` parameter in disk options. Flavor disk_template should be
formated as `ext_<provider_name>`.

16ddd1fe 07/27/2012 07:26 pm Christos Stavrakakis

Add setting indicating whether to use hotplug

3165f027 07/27/2012 02:57 pm Christos Stavrakakis

Implement MAC Prefix Management

In order to guarantee uniqueness of MAC address accross all backends,
synnefo assignes a unique mac prefix to each network in each backend.

A mac-prefix of 20 bits, which is picked from the MacPrefixPool, is
assigned to each Network. The mac-prefix of the network in each...

aeb523a4 07/26/2012 06:29 pm Christos Stavrakakis

Get public network by public attribute

Do not find the public network by the primary key, as the initial_data.json
is obsolete. Instead, ise the public attribute of the Network Model.
Currently only one public Network is supported, and the used can not
create or delete the public Network. But this can easily change in the...

90b29b33 07/24/2012 02:27 pm Christos Stavrakakis

Fix creation of networks in new backend

2b36eda5 07/24/2012 02:26 pm Christos Stavrakakis

Fix bug in process_net_status

Check that nic has an ipv4 before trying to release it.

ad297723 07/23/2012 03:31 pm Christos Stavrakakis

Setting for using IP Pool to public network

Introduce new setting 'PUBLIC_ROUTED_USE_POOL', to indicate whether
synnefo will be responsible for the IP management of the public network,
or this responsibility will be delegated to Ganeti.

967d3f83 07/23/2012 12:16 pm Christos Stavrakakis

Transaction on create_instance

864bed43 07/23/2012 11:49 am Christos Stavrakakis

Implement model manager using row-level db locking

Implement ForUpdateManager that uses SELECT .. FOR UPDATE statement in
order to acquire row-level exclusive locks to DB. Essentially this
manager backports the select_for_update() method that is implemented in...

8a4a931d 07/23/2012 11:49 am Christos Stavrakakis

Fix IP pool exceptions

bd392934 07/23/2012 11:49 am Christos Stavrakakis

Release IP addresses when nics are destroyed

9c65f166 07/23/2012 11:49 am Christos Stavrakakis

Fix error state of BackendNetwork

77f0fa63 07/23/2012 11:49 am Christos Stavrakakis

Implement IP management

Delegate IP management to synnefo, in order to be able to allocate IP
addresses accross multiple Ganeti backends. This is achieved by creating a
Pool of IPs for each Network, in the form of a bitarray, which is stored
in the DB as a base64 string....

839e2bd0 07/23/2012 11:48 am Christos Stavrakakis

Update network settings

0196d9a3 07/23/2012 11:43 am Christos Stavrakakis

Add dirty bit to NetworkInterface

Extend NetworkInterface model with a dirty bit, indicating whether
the index of the NIC is consistent with the one in Ganeti. This flag is
set when deleting a NIC, and is switched off after processing the
notification from Ganeti. Removing a NIC with the dirty bit on is not...

91954b45 07/23/2012 11:42 am Christos Stavrakakis

Revert "Fix bug in dispatcher callback"

This reverts commit 5d40a128fb593ed641400c462f0e00a1b457f76e.

788a54be 07/23/2012 11:42 am Christos Stavrakakis

Remove machine if ganeti job fails

Set the VM deleted flag when receiving a failure notification for a
Ganeti OP_INSTANCE_REMOVE operation, even if the VM operstate is not
ERROR. The server may not exist in the backend not only because the
creation failed, but also because a remove notification was lost.

910f80db 07/23/2012 11:39 am Christos Stavrakakis

Fix bug in dispatcher callback

During handling of ganeti-op-status messages, state was set to error
only if the the opcode was OP_INSTANCE_CREATE. This resulted to
vm's never being deleted, if they did not exist in Ganeti.

36f4cb29 07/23/2012 11:39 am Christos Stavrakakis

Add backend method for removing a vm's nic

22ee6892 07/23/2012 11:39 am Christos Stavrakakis

Modification in logic for new-style networks

b3d28af2 07/23/2012 11:39 am Christos Stavrakakis

Synchronize networks when adding a new backend

fd65ab41 07/23/2012 11:36 am Christos Stavrakakis

Implement BackendNetwork model

Create a new model, to represent the state of each network in each
backend.

cc7c0f44 07/23/2012 11:33 am Christos Stavrakakis

Implement backend allocator and management command

New module implementing instance allocation to one of the available
backends. Allocation is performed based on memory and disk utilization.
Statistics for each backend are stored in DB and updated periodically....

f5b4f2a3 07/23/2012 11:33 am Christos Stavrakakis

Use the Backend client instead of global RAPI

Replace use of global RAPI client, with the cached client from the
VirtualMachine or Backend model.

Dispatch network actions to all backends.

1a894bfe 07/23/2012 11:33 am Christos Stavrakakis

New management commands concerning backends

- Create command to create a backend (snf-manage backend_create)
- Create command to modify a backend (snf-manage backend_list)
- Create command to list backends (snf-manage backend_modify)
- Modify listservers command to display the backend that hosts the VM

924d8085 07/23/2012 11:32 am Christos Stavrakakis

Rename backend_id to backend_vm_id

Rename backend_id field of VirtualMachine model to avoid conflict between the
backend that hosts the VM and the id that the VM will have on the backend.

28f83276 07/09/2012 03:38 pm Kostas Papadimitriou

Merge branch 'simplequotas' into queues+minor_net

ce55f211 07/02/2012 08:25 pm Kostas Papadimitriou

Simple per user quota mechanism for networks/vms

- New dict setting VMS_USER_QUOTA to allow setting per user quota
(overrides MAX_VMS_PER_USER setting)
- Private networks quota mechanism (based on MAX_NETWORKS_PER_USER and
NETWORKS_USER_QUOTA settings)...

c4e55622 06/06/2012 04:56 pm Christos Stavrakakis

Update modules interacting with AMQP

Update all modules which are interacting with the RabbitMQ messaging system to
use the AMQPClient in order to use highly available queues in active/active
mode.

Extend messages to include a unique identifier in order to handle...

1e239ea1 06/05/2012 12:32 pm Dimitris Aragiorgis

Minor change in rapi call

Ganeti 2.5 needs a tupple of length 3 for nics modifications.
So in ModifyInstance rapi call we pass (add, -1, {..}) for nis.

Signed-off-by: Dimitris Aragiorgis <>

3c755209 06/05/2012 12:32 pm Dimitris Aragiorgis

Take advantage of ganeti hotplug

No race condition. No need for ther NetworkInterface to have an
operstate. Hook runs during instance-modify and depicts the currently
existing nics. No reboot needed modification to take place.

Later in mutliple backend mac prefix handling should be done by synnefo....

9c21a2e2 06/01/2012 12:46 pm Christos Stavrakakis

Delete VM nics on update path

Delete VM nics when a message of type OP_INSTANCE_REMOVE has been received
from ganeti, and not when doing the RAPI call.

d1eaa651 06/01/2012 12:41 pm Christos Stavrakakis

Simplify disconnecting vm from network

Use the new option of Ganeti, to remove a specific nic.

7f691719 06/01/2012 12:38 pm Christos Stavrakakis

Use 'network' instead of 'link'

- Rename GANETI_PUBLIC_LINK to GANETI_PUBLIC_NETWORK
- modify create_instance of backend module

2035039b 02/24/2012 05:49 pm Giorgos Verigakis

Switch to the logging middleware