cyclades: Pass network type info as network tag
Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
cyclades: Remove deprecated network type in ganeti
cyclades: Use opportunistic locking
Use opportunistic locking, introduced in Ganeti 2.7, to allow parallel creationof VMs in the Ganeti backend. When this option is set, Ganeti iallocatorwill allocate the VM only to nodes that not locked (e.g. by other VM...
gtools: Fix compatibility issues with ganeti-2.7
cyclades: Fix bug in network reconciliation
Skip reconciliation for a network that does not exist in the DB, ifreconciliation is running without the '--fix' option.
cyclades: Workaround for race condition in Ganeti
Workaround for race where OP_INSTANCE_REMOVE starts executing on Ganetibefore OP_INSTANCE_CREATE. If this is the case, OP_INSTANCE_REMOVE willnot wait for locks, instead it will fail because the instance has not...
Resolve some undefined names
Add missing imports; remove call to sqlite3_changes; set key and valin render_meta.
Revert "wheezy: Remove setup_environ"
This reverts commit a813a25117f3db14d5517f630d6d0e163a58b0f8. Thiscommit is reverted because it breaks compatibility with Django 1.2. Thisshould be applied for Synnefo 0.15.
reconciliation: Strip Ganeti IP addresses
Strip spaces from Ganeti IP addresses, since they are not automaticallyignored in newer versions of ipaddr.
wheezy: Remove setup_environ
From Django docs:""" This function temporarily modified sys.path in order to make the parent“project” directory importable under the old flat startproject layout.This function is now deprecated, as its path workarounds are no longer...
wheezy: Temporary disable reconciliation tests
Temporary disable reconciliation tests because they still run withfixture data, that are not loaded correctly.
cyclades: Simplify code
Simplify code by removing unnecessary temp variable.
cyclades: Do not release resources if already deleted
Ensure that a commission to release vm resources is issued only once.If a vm is marked `deleted', then resources must have already beenreleased.
cyclades: Do not save stale objects after a commit
Call to issue_and_accept_commission saves the resource object (eg a vm)and commits the transaction. This has the side-effect that the resourceobject becomes stale: its lock is released and another thread may acquire...
cyclades: Fix bug in snf-dispatcher logic
This commit reverts commits #07602322 and #30514662 which introduced abug in dispatcher's logic. These commit fixed the issue that, in a casethe Cyclades DB is unsynced with Ganeti, a resource that does not exist...
cyclades: Fix corner case in snf-dispatcher
Same as commit 0514662, but for servers.
Delete network from DB when job fails but the action of network isDESTROY. This case appears when someone tries to remove a network thathas been deleted from Ganeti backend, but not yet from DB.
reconciliation: Get servers from DB ater ganeti
Change server reconciliation, to get the instances from DB after gettinginstances from Ganeti, to decrease the possibility for races.
cyclades: Wrap diagnostic creation in transaction
Fix bug in snf-dispatcher causing wrong updates of VMs from ganeti.Creation if instance diagnostics must run inside a transaction, becauseotherwise, the lock of the vm (select_for_update) is released too early....
Revert "cyclades: Speed up reconciliation"
This reverts commit cf2241c45a22dac71e31dbf32f4700c475804c12.
cyclades: Speed up reconciliation
Speed up reconciliation by using mulitprocessing to get in parallelthe instances from all Ganeti backends.
cyclades: Fix bug in allocating archipelago VMs
This commit fixes a bug that caused backend allocator to sendarchipelago VMs backends to backends not in ARCHIPELAGO_BACKENDSsetting. The flavor disk template is parsed before the backendallocation, so the disk_template at this point is ext and not ext_vlmc,...
cyclades: Speed up server reconciliation
Speed up reconciliation, by increasing the time interval to get Ganetijob status for VMs in building state.
cyclades: Fix deadlock when creating servers
Fix deadlock between snf-dispatcher and backend allocator, by makingsnf-dispatcher take an exclusive lock to the backend which hosts the vm,before trying to get the exclusive locks to the IP pools in order to...
cyclades: Validate MAC prefix in pool-create cmd
cyclades: Pass disk origin for all 'ext' flavors
cyclades: Rename service token
Rename CYCLADES_ASTAKOS_SERVICE_TOKEN to CYCLADES_SERVICE_TOKEN foruniformity.
make synnefo URLs consistent and configurable
https://service.deployment.vendor.tld/service-base/api-prefix/hard/path |___________________________________| |__________| |________| |_______| BASE_HOST BASE_PATH PREFIX Hardcoded...
cyclades: Add 'drained' flag to Network
Extend Network model with the 'drained' attribute, to indicate whether anetwork should be used in public IP allocation when creating servers.This option is only useful for public networks.
cyclades: Export hypervisor to backend-list cmd
cyclades: Add --hypervisor option to backend cmds
Add '--hypervisor' option to 'backend-add' and 'backend-modify'management commands.
Refs #3772
cyclades: Sypport backends with Xen hypervisor
cyclades: Various fixes to tests
dispatcher: Handle NICs in callbacks.
Update 'update_db' callback to handle instance NICs together withinstance status.
Also update the relevant tests.
cyclades: simplify comparison of NICs
Remove 'snf-events-net' queue and binding
Remove 'snf-events-net' queue (together with the corresponding routingkey and binding) from the queues that snf-dispatcher is consumingmessages. This queue is no longer needed, because messages for instanceNICs will be sended by snf-ganeti-eventd to 'snf-events-op' queue....
cyclades: Fix deadlock when connecting vms
Fix deadlock caused by 'get_or_create' method, since it acquires animplicit shared lock. Take an exclusive lock on network to fix thisdeadlock.
cyclades: Fixes relevant to lazy networks
cyclades: Simplify reconcile-networks checks
Refactor and simplify checks of reconcile-networks to take advantageof lazy networks.
cyclades: update tests for lazy networks
cyclades: Update 'backend-add' for lazy networks
Do not create private networks when adding a new backend. Since,"lazy networks" this is no longer needed.
cyclades: Implement lazy networks
Implement "lazy networks", which are not created to all Ganeti backendsupon creation, but only when an instance that resides in a Ganeti isconnected to that network.
Update 'create_network' to create the Network object in DB without...
cyclades: refactor code relative to commission
Refactor Cyclades code that handles commissioning with Quotaholder. Thiscommit removes the 'uses_commission' decorator. Instead it introducesthe 'issue_and_accept_commission' function that is responsible for...
Mock quotaholder for Cyclades tests
Create 'mocked_quotaholder' context manager to bypass commission and useit in Cyclades tests. Also this commit does some various fixes to othertests.
Remove transaction from 'backend-add' command
Do not run 'backend-add' command inside transaction, because if'snf-dispatcher' is running, it will receive messages that will arrivefrom a Ganeti backend that does not exist in the DB and will not be ableto process them.
Move astakos lib from snf-common to snf-django-lib
Use 'python-objpool' instead of 'synnefo.lib.pool'
Use the python-objpool package for pooling resources as object, insteadof the pooling code that existed in snf-common ('synnefo.lib.pool')
Refs #3524
Use pprint_table in server-show command
Create public NIC when creating VM
Until now, a server's public NetworkInterface was not created in the DBuntil the arrival of the message from Ganeti hooks. However, the IPv4address of the NIC is reserved before sending the OP_INSTANCE_CREATEjob to Ganeti....
Move UserCache from cyclades.common to lib.astakos
Move UserCache from synnefo.management.common to synnefo.lib.astakos.Also, update ListCommand with 'astakos_url' and 'astakos_token' classattributes, which must be declared only when 'user_uuid_field' is used.
Move common code for mngmt commands to webproject
Move cyclades functions that are useful for all managament commands, likepprint_table or parse_filters, to snf-webproject.
Add ListCommand to be used for all *-list commands
Create a generic ListCommand to be user for all *-list managementcommands. ListCommand handles must tasks that are common to allmanagement commands, like retrieving objects from db, filteringresults and formating output to a pretty table, json or csv....
Merge branch 'release-0.13' into develop
Conflicts: version
Bypass commissioning for orphan networks
Cyclades public networks do not have an owner, and since can not usethe commissioning system. Also, make snf-dispatcher instead of failingfor networks without userid to just a log a warning. This does not apply for...
Conflicts: snf-cyclades-app/synnefo/api/management/commands/cyclades-export-quota.py snf-cyclades-app/synnefo/api/management/commands/network-list.py snf-cyclades-app/synnefo/api/management/commands/server-inspect.py...
Improve server-list usability
Refs #3375
Improve perf for snf-manage cyclades commands
When the snf-manage {server, network}-list commands are invoked with the--displayname option, batch the UUID-to-displayname requests to astakos,instead of issuing a request for each (unique) UUID.
reconciliation: handle missing network from db
Continue reconciliation even if an orphan network is missing from DB.
Fix typo
pep8 convension
pep8 convension to cyclades management commands
Add --uuids option to snf-manage cmds
Totally remove password from create_instance
Never pass password as an 'osparam' to Ganeti. Always use the vmapi,even for VMs created with server-create management command.
Fix reconciliation for building VMs
Since commit 482c645, a VM exists in the Cyclades DB before thecorresponding job is enqueued in the Ganeti backend. Therefore, there isa corner case, where Cyclades can not enqueue the job to Ganeti andCyclades crashes before deleting the VM from the DB, which results in a...
Remove mac2eui64 dependency
Import mac2eui64 code which is already shipped under util/mac2eui64 insteadof shipping it as a seperate script.
Fix pep8 issues
Fix pep8 warnings
Fix bug in network reconciliation
Fix bug in network reconciliation caused by wrong variable scope.
pep8 fixes
Rename ARCHIPELAGOS to ARCHIPELAGO
Rename ARCHIPELAGOS_BACKENDS setting to ARCHIPELAGO_BACKEND.
Fix progress messages handling
proper handling of missing `messages` key progress messages.
Network field type has been renamed to flavor
Fix typo bug in reconcile ip pools
Add warning message for backend without networks
Refactor network reconciliation
Remove unused imports from tests
Merge branch 'master' into devel-0.13
Conflicts: snf-cyclades-app/synnefo/db/models.py snf-cyclades-app/synnefo/logic/management/commands/backend-modify.py
Refactor tests
Fix bug about networks and offline backends
Ignore offline backends when computing the state of the network, sinceno notification will be received from them. Also, create noBackendNetwork for an offline backend. This will be created byreconcilliation, when the backend gets online again.
Add more tests for db, logic and api
Remove dummy function
Add more tests for db and logic
Refactor some test code
Misc minor bug fixes
Fix current db, logic and api tests
Fix variable scope bug in process_net_status
Do not allow creation of more than one pools
Do not allow creation of more than one Bridge or MacPrefix pool, sincewe can not handle them yet.
Add backend-id option to reconcile-servers
Make reconcile-servers perform reconciliation only for the specifiedbackend.
reconciliation:avoid double asking Ganeti
Modify reconcile-servers to not communicate 2 times with Ganetito get the operstate and the NICs of the VMs. Merge this two onecall.
Modify backend-modify --drained and --offline args
Make backend-modify drained argument as --drained=True|False insteadof --drained and --no-drained. The same for offline.
Mark as drained newly added backends
Rename reconcile command to reconcile-servers
Rename `snf-manage reconcile` management command to `snf-managereconcile-servers`, to be consistent with `snf-managereconcile-networks`. Also, treat '--detect-all' option as defaultwhen no other '--detect-*' is specified.
Introduce network flavors to replace network types
Add tags and mode to network db model. Replace existing type withflavor. Introduce 4 different flavors. Each flavor has mode, link,mac_prefix, tags. Supported flavors are CUSTOM, IP_LESS_ROUTED,MAC_FILTERED, PHYSICAL_VLAN and are hardcoded in db models....
Implement quotas-init and quotas-verify
Implement quotas-init and quotas-verify management commands.- quotas-init sets quotas of Quotaholder to the state of quotas in Cyclades DB- quotas-verify detects incosistencies between quotas in Quotaholder and...
Temporary fix for allocating archipelagos VMs
Fix typos
Merge handling of ext_<provider> disk templates
Merge branch 'origin/devel-0.13' into quota
Conflicts: snf-cyclades-app/synnefo/api/servers.py
Remove transaction from update_network_state
Remove transaction decorator from update_network_state function, sincethis function is only called from process_network_status, which already runsinside a transaction.