backend.py: use hvparams in GetNodeInfo
This patch extends the GetNodeInfo function of the backendmodule by a hvparams parameter. For now, it contains afallback solution for when the parameter is None. Thisfallback will be removed in a later patch of this series....
hypervisors: add hvparams to GetNodeInfo
This patch extends the GetNodeInfo function of thehypervisors by a hvparams parameter. The parameteris currently only used by the xen hypervisor to determinewhich xen toolstack ('xm' or 'xl') to use.
Signed-off-by: Helga Velroyen <helgav@google.com>...
hv_xen.py: renaming residual 'xm' occurrences
Some methods still refered to 'xm', which is inaccurate nowthat xen can also be run with 'xl'. No functional changesotherwise.
Signed-off-by: Helga Velroyen <helgav@google.com>Reviewed-by: Thomas Thrainer <thomasth@google.com>
Include hvparams in ssconf files
The watcher needs access to the cluster's hypervisorparameters. This patch extends the ssconf infrastructureby adding a hvparams file for each hypervisor. The formatof the hvparams file is a simple key=value format.
Use hvparams in instance migration
This patch makes use of hvparams on instance migration todetermine the list of instance before the migration. This way, inxen, the xl/xm choice is respected.
Verify: xen toolstack, hypervisor and hvparams
This patch extends the node verification by:- Adding a check for the xen toolstack when the hypervisor is verified.- Factoring out the hypervisor verification in a subfunction to increase testability.- Factoring out the hvparams verification in a...
backend.py: hvparams in 'GetInstanceList'
Propagating the use of hvparams further up the callinghierarchy, this patch makes the backend module use thehvparams in 'GetInstanceList'. Unit tests provided.
hv_xen.py: _RunXen and GetInstanceList use hvparams
Propagating the use of hvparams further up in the callinghierarchy, the functions '_RunXen' and 'GetInstanceList'use the given hvparams. Unit tests provided. Calling function'StopInstance' adjusted in this patch as well....
hv_xen.py: rename GetXmList to GetInstanceList
Since one can retrieve xen's instance list via both 'xm'or 'xl', it is no longer appropriate to call the function'GetXmList'. This patch renames it to 'GetInstanceList'.Other than that, there are no functional changes....
hv_xen._GetCommand: retrieve xen command from hvparams
This patch adds a (so far optional) hvparams parameterto the '_GetCommand' function. This dictionary is usedto retrieve the xen command (xm or xl). It is optionalfor now to make the refactoring possible without breaking...
Unit tests for hv_xen._GetCommand
This patch adds unit tests for the current state of the'GetCommand' method. The tests are rather trivial, buthaving tests before starting to refactor seems to bea good idea.
Fixing docstring of hv_xen_unittest.py
The introductory comment of hv_xen_unittest.py referred tothe wrong hypervisor.
Export node spindles
Node spindles (queried live) are exported through the LUXI, RAPI, andiallocator interfaces.
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>Reviewed-by: Klaus Aehlig <aehlig@google.com>
Check real spindles in ipolicies
When exclusive storage is enabled, the spindles in instance disks are usedto check the instance policies (as outlined in design-partitioned.rst).
Merge branch 'stable-2.8' into master
Make the disks_active flag queryable
gnt-instance list now also supports the disks_active field.
Signed-off-by: Thomas Thrainer <thomasth@google.com>Reviewed-by: Helga Velroyen <helgav@google.com>
Attaching a logical volume builds a list of PVs
When an LV gets attached, the list of the PVs used by the LV is built. Thiswill be used to count spindles for exclusive_storage, but it could also beuseful to optimize disk growing and snapshotting.
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>...
Refactor code for attaching a logical volume
The parsing of "lvs" output is moved into private methods. The code isslightly more readable and testable. The split in two methods is usefulfor the following patches. Unit tests for the new functions areprovided....
Merge branch 'stable-2.8'
cfgupgrade: Downgrade is a NO-OP
The configuration is still the same as in 2.8 (the reference stable versionfor this branch), so downgrade shouldn't do anything.
Unit tests are also updated, with a new 2.8 configuration file. Theconfiguration file used for the upgrade+downgrade test was tailored to the...
Stash Xen config file after a failed startup
This is a fix for a minor bug.Currently, a failed Xen VM start results in a stale config file leftbehind on the filesystem.This change introduces a new log directory, where the Xen VM configfile is moved after a failed startup....
cmdlib: Cleanup public/private functions
All functions/classes which are used outside of their defining module(with tests as an exception) no longer have a leading underscore.
Signed-off-by: Thomas Thrainer <thomasth@google.com>Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>
cmdlib: Extract storage related functionality
Split instance.py further by extracting storage related logical unitsand functions to instance_storage.py.
Extract query related logical units from cmdlib
All LUQuery* classes are extracted to query.py.
Extract instance related logical units from cmdlib
All LUInstance* classes are extracted to instance.py. Common functionsare moved to common.py if used by non-instance logical units as well.Additionally, helper functions which are only used by LUBackup* and...
Extract node related logical units from cmdlib
All LUNode* classes are extracted to node.py. Common functions are movedto common.py if used by non-node logical units as well.
Extract group related logial units from cmdlib
All LUGroup* classes are moved to group.py. Common functions areextracted to common.py.
Extract cluster related logical units from cmdlib
All LUCluster* classes are extracted to cluster.py. Shared functions areextracted to common.py, helper functions only used by LUCluster* areextracted to cluster.py.
Signed-off-by: Thomas Thrainer <thomasth@google.com>...
Use os.statvfs to determine free disk space
This simplifies my previous commit (820bade90) by using os.statvfsinstead of parsing the output of 'df'.
Signed-off-by: Helga Velroyen <helgav@google.com>Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>
Backend function for file storage space reporting
This adds functionality to retrieve disk space informationfor file storage. It calls the 'df' tool and parses itsoutput to extract the total and free amount of disk spaceon the disk where the given path is located....
Move 'container.py' to storage directory
Moving 'container.py' to the storage directory.
Rename dir 'block' to 'storage'
Renaming the 'block' directory to 'storage', because I plan toplace code there that is related to file storage and leavingit named 'block' would be misleading.
Rename storage.py to container.py
Renaming 'storage.py' to 'container.py'. It will be moved into the new'storage' directory, which will come in later patches to avoid clashes ofnotation.
Split DRBD8Dev into DRBD8 and DRBD8Dev
All functionality specific to a single DRBD8 devide is now in DRBD8Dev,whereas functionality which is valid for the whole DRBD "installation" on a device is collected in DRBD8.
This makes it possible to remove a couple of FIXME's and clarifies the...
Rename DRBD8 to DRBD8Dev
Right now the DRBD8 class has multiple responsibilities: a) itreprensents a device which can be set up, grown, etc. and b) itrepresents the whole DRBD system on a node which has a usermode helper,which knows how to shut down all devices, etc. Therefore, the DRBD8Dev...
Make DRBD version queryable from noded
gnt-cluster verify should issue a warning if there are multiple DRBDversions present in a node group. In order to do so, the DRBD versionhas to be queryable from noded.
Merge remote-tracking branch 'origin/stable-2.8'
Conflicts: lib/bdev.py lib/cmdlib.py
Signed-off-by: Michele Tartara <mtartara@google.com>Reviewed-by: Klaus Aehlig <aehlig@google.com>
Disk templates are "allowed" and not "enabled" in ipolicies
The way disk templates were displayed in instance policies became confusingsince the introduction of the enabled_disk_templates cluster parameter.
Fix issue 440.
Unit test for FillIPolicy keeping unknown keys
This makes sure that issue 401 doesn't come back.
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>Reviewed-by: Helga Velroyen <helgav@google.com>
Add command generator for DRBD 8.4
The DRBD84CmdGenerator class, which generates commands suited for DRBD8.4, is added. A common baseclass for DRBD83CmdGenerator andDRBD84CmdGenerator is introduced as well.
Add test for version dispatching code
Based on the version of DRBD (as reported through /proc/drbd), the DRBD8class chooses helper objects which adapt for the differences betweenDRBD <= 8.3 and DRBD 8.4. This new test verifies that the proper objectsare created....
Extract DRBD related tests
As the DRBD code was extracted from bdev.py, the corresponding testsshould as well be extracted into their own file. That's what this patchperforms.
Signed-off-by: Thomas Thrainer <thomasth@google.com>Signed-off-by: Michele Tartara <mtartara@google.com>...
Extract command generation logic from DRBD8
In order to easily support the DRBD 8.4 command syntax, this patchextracts all command generation logic (i.e. where calls to `drbdsetup`are assembled) into a new class DRBD83CmdGenerator.
Add `drbdsetup show` parser for DRBD 8.4
Common functionality between the DRBD 8.3 and DRBD 8.4 parser has beenextracted into BaseShowInfo. A test which verifies the behaviour isincluded, but the DRBD84ShowInfo class is not yet used within the DRBD8class....
Extract DRBD info parsing into drbd_info.py
Parsing the status (/proc/drbd) and `drbdsetup show` output takes quitea bit of code, which has to be augmented by even more with the DRBD 8.4support. So extract all the related classes into their own file.
Extract DRBD8ShowInfo class
This class parses the `drbdsetup show` output and represents it in aneasily accessible format. It got extracted so that 1) the DRBD8 classcan focus more on DRBD logic rather than parsing and 2) it's easier toadapt the parser to new formats....
Add test for DRBD 8.4 /proc/drbd parsing
The current parser already supports the output of DRBD 8.4 /proc/drbd,so add tests which actually verify this behaviour.Note that the empty lines in proc_drbd84.txt are actually present in theoutput of DRBD 8.4, they always appear when minors are not contiguous....
Add test for empty version in /proc/drbd
The previous patch removed a (rather implicit) test for /proc/drbd notcontaining a version line. This adds an explicit test for this case.
Extract /proc/drbd parsing code into DRBD8Info
As the DRBD8 class got bigger due to the previous merge of BaseDRBD, nowparts of it are ripped out into DRBD8Info. This new class parses/proc/drbd and exposes the information in an easily accessible way. This...
Remove BaseDRBD
BaseDRBD was probably useful when DRBD 0.7 and DRBD 8 were supported.However, there is only one subclass of BaseDRBD remaining (DRBD8), andthe separation of responsibilities between those two classes was ratherrandomly chosen.
The unification into one class also eases the introduction of...
Rename _MassageProcData to _JoinProcDataPerMinor
That's what the method actually does, so there is no reason why weshould use a funny but unreadable name.
Unit test for cli.FormatPolicyInfo()
We try to parse the string obtained by formatting a policy, and see if weget the same policy.
Add command-line support for multiple specs in ipolicy
Command line options accept multiple min/max specifications in instancepolicies. The output of show-ispecs-cmd is updated accordingly.
Add multiple min/max specs in instance policy
Now instance policies can contain more than one min/max specs. This is themain element of the "Constrained instance sizes" section in the"Partitioned Ganeti" design doc.
This is a big patch, but changing the type of a configuration item requires...
Separate checks for std spec compliance
This is needed to be able to validate the std spec against multiple min/maxspec pairs (appearing in next patches).
Merge branch 'stable-2.7' into stable-2.8
Conflicts: NEWS doc/rapi.rst
Signed-off-by: Klaus Aehlig <aehlig@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Reason trail implementation for "start"
Opcode-specific implementation of the reason trail for the instancestartup operation.
Signed-off-by: Michele Tartara <mtartara@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Reason trail implementation for "shutdown"
Opcode-specific implementation of the reason trail for the instance shutdownoperation.
Add unit tests for cfgupgrade with a real configuration
Test upgrade and downgrade from a realistic 2.7 configuration.
Split functions in cfupgrade unit tests
This makes easier to test real configurations.
Limit specs in ipolicy command lines are always complete
Command line options are brought in line with the specs change of previouspatch. Old options are still allowed in gnt-cluster init, where thesemantic will remain non-ambiguous even after introducing multiple specs....
Refactor tests in ipolicy validation
This makes the code more modular, in preparation for the following patches.
Limit specs in instance policies are always complete
Specs used to specify limits in instance policies are always complete, evenat group level (only the whole limit set can be overridden). This is inpreparation for introducing multiple limits.
Add --ipolicy-xxx-specs options
These options allow to specify whole instance policy specs. This is neededfor the upcoming changes that tend to threat specs as monolithic objects.
Add command to print ipolicy options
The output of this command can be used to create an exact copy of thecurrent instance policy specs. The command could be expanded to print allthe options used to create a group or the cluster.
New CLI input type: list of key/value pairs
This will be used for the new instance specs options.
Allow "default" as a value for ipolicy templates
Previously the string value was being corrupted.
Extract DRBD-related classes into block/drbd.py
This is in preparation to introducing support for DRBD8.4.base.py had to be extracted as well in order to avoid cylic importsbetween bdev.py and drbd.py. It now contains the BlockDev class andutility functions needed by bdev.py and drbd.py....
Move lib/bdev.py to lib/block/bdev.py
That's in preparation of extracting DRBD related code from bdev.py. Asbdev.py is already rather long, new features will require to split itinto more manageable pieces. That's why it's moved in an own directory.
Reason trail implementation for instance reboot
This commits allows ganeti to correctly forward the reason trail informationregarding instance reboot.
Signed-off-by: Michele Tartara <mtartara@google.com>Reviewed-by: Helga Velroyen <helgav@google.com>
Remove old "reason" implementation
Remove the useless parts of the old, partial, implementation of the support fortracking the reason of instances state change, before implementing the newreason trail support, as per the design document.
Signed-off-by: Michele Tartara <mtartara@google.com>...
Update QA tests to check disk and NIC names
Modify existing tests to use the name option for instance disks. Theconfiguration options 'disk' and 'disk-growth' are merged into a single'disks' option, which is a list of dictionaries with 'size', 'growth' and...
Check that device names are unique and valid
Extend the CheckArguments phase of LUInstanceCreate and CheckPrereqphase of LUInstanceSetParams to also check if the name parameters ofdisks and NICs are unique and valid.
Signed-off-by: Christos Stavrakakis <cstavr@grnet.gr>...
Support UUIDs and names when refering to a device
Modify _ApplyContainerMods function to lookup NICs/Disks not only by theirindex inside the container, but also by their UUID or name. Abstract thelookup code in new GetItemFromContainer function.
Make type of identifier in "opcode._TestInstSetParamsModList" to be...
Move HooksMaster out of the mcpu module
We need to do this, so that backend.py doesn't need to import mcpu, andthus indirectly cmdlib. This reduces the size of the node daemon byabout half, which is very important as it is pinned in memory.
This solves Issue 419....
QA: Decision about disk templates to use is more modular
qa_config.IsTemplateSupported() is already used to decide which disktemplates are supported by QA. Enabled templates now are part of thatdecision.
Unit tests are updated.
Unit tests for config.ConfigWriter.VerifyConfig()
The test is aimed mostly at instance policies, which changed recently.
Refactor ispecs in ipolicy structures
Minimum and maximum instance specs are put together into a single elementof the instance policy. This is in preparation for introducing multiplemin/max specs.
cfgupgrade: Add --downgrade option
It's now possible to go back to the previous stable version. Unit testsprovided.
This is mostly useful during development, when going from master tostable/devel.
Unit test for cli.CreateIPolicyFromOpts()
Useful for subsequent patches.
Generic function to print data for gnt-xxx info
The function produces a human-readable output, which is also a valid YAMLfile, from an intermediate data structure. This will be used by "gnt-xxxinfo" commands. The output of "gnt-xxx info" was almost YAML-compliant, and...
Fix bug in rlib2 unit tests
The "queryargs" parameter of the init function of rlib2 classes shouldbe a dictionary, as defined in the ResourceBase parent class (inlib/rapi/baserlib.py).
In the rlib2 and baserlib unit tests, when the queryargs parameter is not...
Add the core of the monitoring daemon
This commit adds the core infrastructure of the monitoring daemon,and integrates it in the build and test systems.
The actual functionality of the monitoring daemon is still completelymissing.
Add infrastructure for allowing additional logfiles
Some daemons will need more than the single logfile that is currentlyallowed. This patch introduces the infrastructure to allow this.
Signed-off-by: Michele Tartara <mtartara@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Make gnt-node list -o(p|s)inst_list output stable
Currently, both the Python and Haskell code return the internalinstance list unsorted, which means the output can vary depending onthe phase of the moon (well, the Haskell code actually uses internallya tree, sorted by the instance name, but it's implementation detail)....
Merge branch 'devel-2.7'
Unit tests for objects.InstancePolicy + a fix
Tests for: objects.InstancePolicy.CheckParameterSyntax() objects.InstancePolicy.CheckDiskTemplates() objects.InstancePolicy.CheckISpecSyntax()
Instance policies with an empty disk-template list now are reported....
Unit tests for objects.FillIPolicy() + small fix
IPOLICY_DEFAULTS is now a legal policy (the disk-templates entry was a setinstead of a list, before).
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Fix upgrade of policy in objects.Cluster
Unknown elements were silently removed on startup. This means that asoftware upgrade could result in lost configuration information ifcfgupgrade wasn't run promptly.
Added unit test for Cluster.UpgradeConfig() to cover this case....
Fix instance policy checks for default back-end parameters
Policy violations of back-end parameters that used the cluster defaultvalue were not reported in cluster-verify.
Unit tests for cmdlib._GetUpdatedIPolicy()
Not 100% coverage, though.
Fix policy check for disk templates
Instance disk template is checked against the policy, and disklessinstances aren't checked for the number of disks.
Allow rpc.MakeLegacyNodeInfo to parse non-LVM results
'MakeLegacyNodeInfo' is not the best place for this, but we'd have toduplicate it if we wanted a LVM-less version, so the easiest is to addan optional parameter that allows it to accept/skip LVM-less results....
Test AsyncStreamServer with abstract unix sockets
This was meant to be since "the beginning" but was never submitted assomehow it failed with python 2.4. Now that the minimum python versionhas been increased it can be added.
Signed-off-by: Guido Trotter <ultrotter@google.com>...
qa_config: Support callable test checks
Until now all tests depended on simple AND/OR conditions. With theintroduction of virtual cluster support, some tests can only run ifvirtual clusters are not in use (e.g. not yet supported or notpossible). Instead of introducing more logic for enabling/disabling...
QA: Add config entries for virtual cluster
If these new options are set, a virtual cluster is created and used.Virtual clusters consist of 1..N virtual nodes on a single physical (orvirtual) node. The design of virtual clusters is described in“doc/design-virtual-clusters.rst”....
Fix style violations in commit 4a90bd4, small changes
- Commit 4a90bd4 contained a rather large number of style violations: - Indentation/sequence formatting - Wrapping of long lines - Docstrings whose first line was wrapped - A stray backslash in a docstring...