Merge branch 'stable-2.10' into master
Add a Python directory for RPC code to keep it at one place
Move rpc.py to rpc/node.py and modify imports in existing code.
Signed-off-by: Petr Pudlak <pudlak@google.com>Reviewed-by: Helga Velroyen <helgav@google.com>
Gluster: minimal implementation
Add Gluster to Ganeti by essentially cloning the shared file behavioureverywhere in the code base.
Signed-off-by: Santi Raffa <rsanti@google.com>Signed-off-by: Thomas Thrainer <thomasth@google.com>Reviewed-by: Thomas Thrainer <thomasth@google.com>
Merge branch 'stable-2.9' into stable-2.10
Merge branch 'stable-2.8' into stable-2.9
Fix evacuation out of drained node
Add default_iallocator_params cluster parameter
Add a cluster parameter to hold the iallocator parameters usedby the default instance allocator. Implement the option tomodify config.data, query config.data and upgrade man pages,tests and cfgupgrade tool. The new default_iallocator_params is...
Remove FIXME + update man pages wrt iallocator
This patch removes a FIXME regarding the retrievalof the disk template for computation of the clusterinformation for the iallocator. After examining thecode, I figured out that it is actually a quiterobust way to just check if the disk template is...
Add possibility to compress to OpInstanceCreate
OpInstanceCreate now supports the 'compress' option. It allows to enablecompression during instance imports.
Signed-off-by: Thomas Thrainer <thomasth@google.com>Reviewed-by: Klaus Aehlig <aehlig@google.com>
Honor compress parameter in remote backups
Use the specified compression mode not only for local exports, but alsofor remote ones.
Add local compression to OpBackupExport
OpBackupExport is extended by a compress parameter. This parameter(either 'none' or 'gzip') controls if instance disks are compressedbefore being sent over the network to the destination node.
Signed-off-by: Thomas Thrainer <thomasth@google.com>...
Add possibility to compress to OpInstanceMove
OpInstanceMove now supports the 'compress' option. It allows to enablecompression for intra-cluster instance moves.
Streamlining handling of spindles and default templates
This rather lenghy patch comprises a couple of refactoringsto achieve the following goals:- 'gnt-node info' should only report space information about spindles, when exclusive storage is enabled on the node...
iallocator: use lookup by disk template
So far, the iallocator requested storage informationabout all enabled disk templates but discarded all butthe LVM information, even if it was allocating spacefor a non-LVM instance. With this patch, it now onlyasks for the one that is relevant for...
Merge 'EvacNode' and 'NodeEvacMode'
Merge 'EvacMode' and 'NodeEvacMode' which are associated with theduplicated constants 'IALLOCATOR_NEVAC_*' and 'NODE_EVAC_*'. Thismerge moves the type 'EvacMode' to 'Ganeti.Types' in order to avoid acircular dependency between 'Ganeti.HsConstants' and...
Replace 'IALLOCATOR_NEVAC_*' with 'NODE_EVAC_*'
Replace uses of 'IALLOCATOR_NEVAC_PRI', 'IALLOCATOR_NEVAC_SEC','IALLOCATOR_NEVAC_ALL', and 'IALLOCATOR_NEVAC_MODES', with'NODE_EVAC_PRI', 'NODE_EVAC_SEC', 'NODE_EVAC_ALL', and'NODE_EVAC_MODES', given that these constants are repeated....
Display node names instead of node UUIDs
Display resolved node names instead of node UUIDs in a couple ofwarnings and log messages.
Signed-off-by: Thomas Thrainer <thomasth@google.com>Reviewed-by: Jose A. Lopes <jabolopes@google.com>
Remove physical_id field from disk object
The 'physical_id' field of disk objects is no longer used, so remove it.Also, all references are removed together with the code which made surethat the physical_id is up to date when transmitted over RPC.
Replace physical_id with dynamic_params
The disk field 'physical_id' has to be kept up to date whenever a diskobject is sent to a node via RPC. This is done with the SetDiskID methodmanually, which is a source of bugs.
This patch replaces the use of 'physical_id' with a new field names...
Fixing renaming of DISK_LD_DEFAULTS
This fixes the somewhat over-eager renaming ofDISK_LD_DEFAULTS to DISK_DT_DEFAULTS, which clasheswith another occurence of DISK_DT_DEFAULTS. Thispatch just undoes the renaming and additionally fixesa test regarding shared file storage....
Replace LD_* constants with DT_* constants
LD_* constants are basically like DT_* constants, exceptfor that both file and shared file were mapped to file.In order to not having to maintain three slightly differentsets of disk-related constants (DT, LD and ST), we merge...
Fix typo in iallocator
Signed-off-by: Helga Velroyen <helgav@google.com>Reviewed-by: Klaus Aehlig <aehlig@google.com>
Extend RPC call 'node_info' by storage parameters
This patch extends the actual RPC call to accept storageparameters with each storage units. It adjusts all codewhich performs this code by integrating the 'exclusivestorage' flag into the lvm storage unit....
Export CPUs used by the node OS
They are exported through the LUXI, RAPI, and IAllocator interfaces.
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>Reviewed-by: Klaus Aehlig <aehlig@google.com>
iallocator: storage processing accepts space_info structure
This is a small refactoring to make the processing ofthe storage space information not accept a node inforesult, but the more specific space info structure instead.
Signed-off-by: Helga Velroyen <helgav@google.com>...
iallocator: get rid of MakeLegacyNodeInfo
So far, the iallocator makes a node info call, convertsthe result into a legacy node info structure and thenretrieves various values from it. This patch removesthe call to MakeLegacyNodeInfo completely, by letting the...
iallocator: factor out instance memory summation
The iallocator adjusts the free memory by a calculationof how much memory is actually currently used by instanceson the node. This calculation is rather complicated anddeserves its own function to increase testability. No...
iallocator: extract storage info directly from node info
So far, the iallocator made a node info RPC call, convertedthe result into a legacy node info format and then extractedthe storage information from that. This patch removes themiddle step of this procedure, skipping the legacy node info...
iallocator: computation of storage data
This patch extracts the processing of the storage-relateddata from the RPC call 'node_info' in a separate functionto increase testability. In addition, this patch containsunit tests for this function.
iallocator: factor out 'get_attr' method
This patch extracts an inline 'get_attr' method to extractdata from the result of the RPC call 'node_info'. It was sofar an inline function in the middle of a bigger method, which madethe code hard to read.
iallocator: prepare RPC call 'node_info'
This patch does a small refactoring in the allocator. It extractsthe preparation and actual call of the 'node_info' call into aseparate function. This is done for better testability. No functionalchanges otherwise....
Index instances by their UUID
No longer index instances by their name but by their UUID in the clusterconfig. This change changes large parts of the code, as the followingadjustments were necessary: * Change the index key to UUID in the configuration and the...
Refactor NodeInfo RPC regarding storage reporting
The NodeInfo RPC call is refactored to handle now more thanjust storage reporting for volume groups.
Since NodeInfo now returns storage space information notnecessarily for volume groups, but also for other storage...
Inline simple "alias" variables
Remove variables in the form "var = self.var" throughout all LU's. Those"alias" variables make it harder to read the code (it's no longerimmediately clear what's a field and what's a local variable), and canintroduce subtle bug (especially when not used consistently and when...
Index nodes by their UUID
No longer index nodes by their name but by their UUID in the clusterconfig. This change changes large parts of the code, as the followingadjustments were necessary: * Change the index key to UUID in the configuration and the ConfigWriter, including all methods....
Use hvparams in GetAllInstancesInfo
This patch extends the GetAllInstancesInfo method of thehypervisors by a hvparams parameter. This change propagatesup to the RPC call all_instances_info, which also neededto be extended. All callers of this call were adjusted...
Adjust all callers of RPC 'call_node_info'
This patch contains all adjustments necessary to code thatuses the 'call_node_info' call. The adjustment is necessarybecause the call's signature changed from receiving a listof hypervisor names to a list of tuples...
Export node spindles
Node spindles (queried live) are exported through the LUXI, RAPI, andiallocator interfaces.
Export instance disk spindles
Instance disk spindles can be queried via LUXI and RAPI, and are exportedthrough the allocator interface. This is a prerequisite for htools tohandle spindles.
The length of a RAPI query including all the instance fields now has become...
Merge branch 'stable-2.8' into master
Export connected networks to IAllocator
IAllocators should not put instances on nodes in nodegroups which arenot connected to the network the instance should be connected to.Therefore, export the networks a node group is connected to to theIAllocator, so it can account for this restriction....
Add disks_active to configuration
This flag tracks if the disks of an instace are supposed to be active.That's the case when an instance is running or when its disks gotactivated explicitly (and in a couple of other cases).It will be used by watcher to re-activate disks after a node reboot....
RPC 'node_info': <storage_type,key> instead of vg_names
This replaces the field 'vg_names' in the RPC call of 'node info' by'storage_units'. A storage unit is a tuple <storage_type,key>and a generalization of a vg_name. The list of vg names is replaced by...
The disk size of a diskless instance is 0, not None
For diskless instances it is still reasonable to sum up the disk usageof all the (zero) disks, resulting in the empty sum. This uniformityalso has the advantage that iallocators (like hail) do not have to do...
Allow iallocator to work without LVM storage
Currently, the iallocator interface requires LVM storage, due to theway it computes the node storage information.
By changing the code to understand that GetVGName() can return None,and by setting the disk_total/disk_free node parameters to the value...
Rename lib/objectutils to outils.py
Back when this was introduced, I mentioned that it breaks heavily tabcompletion (ob<TAB> doesn't work anymore), but at that moment I didn'thave a suggestion what to name it. I think outils is good and shortenough, and doesn't conflict with anything else, so here it goes....
Call node_info RPCs with the exclusive_storage flag
The flag is read from the configuration and passed to the RPC.
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Add exclusive_storage parameter to node_info RPC
The flag is passed to the low-level functions that need it, but it's notyet used. Also, RPCs get just a default value for now.
Fix iallocator for nodegroups with no tags
_ComputeNodeGroupData() raises error if a group has no tagswhen list(gdata.tags) is invoked. Use “gdata.GetTags()” instead.
Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>Signed-off-by: Michael Hanselmann <hansmi@google.com>...
Implement the External Storage Interface
With this commit we introduce the External Storage Interfaceto Ganeti, abbreviated: ExtStorage Interface.
The ExtStorage Interface provides Ganeti with the ability to interactwith externally connected shared storage pools, visible by all...
iallocator: Add node whitelist
In the future instance creations might have a lock on all nodes as wasthe case until the implementation of opportunistic locking. Nodes forwhich the lock is not held will be shown to the iallocator plugin as ifthey were marked offline....
Fixes to pass pep8 (make lint)
Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>Reviewed-by: Iustin Pop <iustin@google.com>
Cleanup ht's use of positive/strictpositive
Currently, ht.py uses a bad terminology for positive/non-negativenumbers. Per http://en.wikipedia.org/wiki/Positive_number, this is thecorrect terminology:
- A number is positive if it is greater than zero.- A number is negative if it is less than zero....
Merge branch 'devel-2.6' into master
Verify the node count just if the allocation was successful
Signed-off-by: René Nussbaumer <rn@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
Fix REQ_RESULT for MultiInstanceAllocRequest
Fix a validation issue
We want to validate the result only if the iallocation run was asuccess.
Signed-off-by: René Nussbaumer <rn@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>
Migrate lib/masterd/instance.py from constants to pathutils
File system paths moved from constants to pathutils.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Small style adjustment
Make iallocator request types more strict
This adapts the type of some request input fields to more stricter ones.
Rework the required_nodes field
This was used to verify the result of instance allocation. However, thisis not logic which belongs to the cmdlib rather than to the iallocatorrequest itself. So abstract it by its own function. And update theValidateResult accordingly....
Disable pylint warnings for unknown members
For some reason pylint can't determine, that the members are calculatedupon runtime and complains. This is weird as the same mechanism works onopcode/objects. I haven't found the issue for it yet, so let's just...
Adding new multi-allocation request
Now that we've the refactoring done, we can easily add the new mode andeverything works.
Refactor IAllocator code
The IAllocator class was handling all the requests on its own, passingin parameters on top level which works, but is hard to maintain and notflexible.
With the upcoming change to the IAllocator for MultiAllocate we can'tuse the toplevel parameters anymore. Therefore, we refactor the code...
Move _CalculateGroupIPolicy and _ComputeDiskSize
This makes it possible to better untangle the IAllocator code, whichwould otherwise lead to a recursive import structure as we need thosehelpers there as well.
Signed-off-by: René Nussbaumer <rn@google.com>...
cmdlib: Adapt the rpc calls
The following (blockdev) RPC calls are not converted yet (as they arenot straight forward or need more research):
Add basic support for disk parameters
objects.py: * add disk parameters to Disk, Cluster, NodeGroup.
constants.py: * add dictionaries that will hold types and default values for disk parameters (for now, empty).
test/ganeti.constants_unittest.py:...
rpc: Change {import,export}_start to take source/dest in single argument
This simplifies the RPC argument encoding code.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: René Nussbaumer <rn@google.com>
Move _TimeoutExpired to utils
Signed-off-by: Andrea Spadaccini <spadaccio@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
Instance transfer: export component name to backend
This modifies the RPC layer to export the component name too to thebackend, so that it can be used in log files and messages.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
Instance transfer: add argument for the 'component'
Currently, transfer data is done mainly with just the instance name,but when we have instances with multiple disks this is not enough todistinguish between the different transfers being done for theinstance....
Replace %r with '%s' in masterd/instance.py
I still don't know why Michael is a fan of %r, but in the meantimethis patch changes:
WARNING: import u'import-2011-07-29_01_39_33-y3gZKV' on node1 failed:Exited with status 1
into:
WARNING: import 'import-2011-07-29_01_39_33-y3gZKV' on node1 failed:...
Import-export: fix logging of daemon output
In case of failures, the recent daemon output is logged as %r on alist of unicode strings, which results in the (ugly):
Thu Feb 3 05:13:34 2011 snapshot/0 failed to send data: Exited with status 1 (recent output: [u' DUMP: Date of this level 0 dump: Thu Feb 3 05:13:18 2011', u' DUMP: Dumping /dev/mapper/6369a5f7-1e67-4d0d-a4f0-956b3649c6d7.disk0_data.snap-1 (an unlisted file system) to standard output', u' DUMP: Label: none', u' DUMP: Writing 10 Kilobyte records', u' DUMP: mapping (Pass I) [regular files]', u' DUMP: mapping (Pass II) [directories]', u' DUMP: estimated 54301 blocks.', u' DUMP: Volume 1 started with block 1 at: Thu Feb 3 05:13:19 2011', u' DUMP: dumping (Pass III) [directories]', u' DUMP: dumping (Pass IV) [regular files]', u'socat: E SSL_write(): Connection reset by peer', u"dd: dd: writing `standard output': Broken pipe", u' DUMP: Broken pipe', u' DUMP: The ENTIRE dump is aborted.'])...
Merge branch 'devel-2.3' into devel-2.4
Improve import/export timeout settings
With this patch, the exporting node will retry to connect a few times.The receiving node will make use of the master's increased timeout (seeprevious patch).
Make snapshots multi-vg aware
Currently, the Snapshot() function of LogicalVolume returns only thelogical volume path, with the assumption that we only have one VG. Butwith the recent changes, it makes more sense to return the full data (vgand lv) from it, so as to not require computing it in the master....
Use IPv6 for import/export
Support for resolving hostnames to IPv6 addresses
This patch enables IPv6 name resolution by using socket.getaddrinfoinstead of socket.gethostbyname_ex.
It renames the HostInfo class to Hostname and unifies its use throughoutthe code. This is achieved by using static calls where no object is...
masterd.instance: Add missing argument
_DiskTransferPrivate takes three parameters, not two.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Manuel Franceschini <livewire@google.com>
Introduce lib/netutils.py
This patch moves network utility functions to a dedicated module.
Signed-off-by: Manuel Franceschini <livewire@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Remove pred from compat.any/all
This makes it compatible with the python builtin, and we can even usethe builtin when running under the right version of python. The all andany functions are renamed to _all and _any, so that they can be tested,and (non)existing unittests are updated (translation: there are no unit...
Use import/export magic for backup/import and inter-cluster moves
This should prevent bugs in our code from accidentally overwritingdisks.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Disable compression for all intra-cluster imports/exports
Tests have shown that usually we're CPU-bound for intra-clusterimports/exports. Disabling compression will help with this.
Some versions of OpenSSL, depending on the build options, alsocompress transparently. This will need further work in Ganeti....
backend: Add support for import/export magic
import/export: Validate remote host/port
The hostname and port received from the remote cluster shouldbe validated, just in case.
Show formatted ETA for disk sync and import/export
import/export: Show progress updates to user
With this patch, we show progress updates approx. once per minute.
Allow control of import/export compression method
For exports to/imports from the same machine, compression willnot be used anymore.
Put common import/export daemon options into object
The X509 key name and CA are passed from cmdlib all the way tothe backend import/export daemon. With the addition of an optionto choose the compression method, another parameter would haveto be passed all the way. By moving these options to a separate...
Implement opcode changes for remote-import
Implement opcode changes for remote-export
Add opcode to prepare export
To prepare a remote export, the X509 key and certificate need to be generated.A handshake value is also returned for an easier check whether both clustersshare the same cluster domain secret.
Signed-off-by: Michael Hanselmann <hansmi@google.com>...
LUExportInstance: Move exporting code to helper class
This will simplify the implementation of intra-cluster instanceexports and reduces the number of local variables inLUExportInstance.Exec.
cmdlib: Add utility to transfer instance data within the cluster
This is yet another wrapper around the instance import/export utilityclasses, providing an even simpler API for instance imports/exports withinthe same cluster.
cmdlib: Add utility for instance data import/export
Interpreting the backend's import/export daemon status is a bit tricky.This utility code keeps track of multiple transfers at the same time.Users can supply callback functions to react to events.
Timeouts are currently hardcoded. Intra-cluster instance moves will likely...
Add ganeti.masterd module
This will be used to keep masterd-specific code.