Use os.statvfs to determine free disk space
This simplifies my previous commit (820bade90) by using os.statvfsinstead of parsing the output of 'df'.
Signed-off-by: Helga Velroyen <helgav@google.com>Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>
Backend function for file storage space reporting
This adds functionality to retrieve disk space informationfor file storage. It calls the 'df' tool and parses itsoutput to extract the total and free amount of disk spaceon the disk where the given path is located....
Move 'container.py' to storage directory
Moving 'container.py' to the storage directory.
Signed-off-by: Helga Velroyen <helgav@google.com>Reviewed-by: Thomas Thrainer <thomasth@google.com>
Rename dir 'block' to 'storage'
Renaming the 'block' directory to 'storage', because I plan toplace code there that is related to file storage and leavingit named 'block' would be misleading.
Rename storage.py to container.py
Renaming 'storage.py' to 'container.py'. It will be moved into the new'storage' directory, which will come in later patches to avoid clashes ofnotation.
Split DRBD8Dev into DRBD8 and DRBD8Dev
All functionality specific to a single DRBD8 devide is now in DRBD8Dev,whereas functionality which is valid for the whole DRBD "installation" on a device is collected in DRBD8.
This makes it possible to remove a couple of FIXME's and clarifies the...
Rename DRBD8 to DRBD8Dev
Right now the DRBD8 class has multiple responsibilities: a) itreprensents a device which can be set up, grown, etc. and b) itrepresents the whole DRBD system on a node which has a usermode helper,which knows how to shut down all devices, etc. Therefore, the DRBD8Dev...
Make DRBD version queryable from noded
gnt-cluster verify should issue a warning if there are multiple DRBDversions present in a node group. In order to do so, the DRBD versionhas to be queryable from noded.
Signed-off-by: Thomas Thrainer <thomasth@google.com>...
Merge remote-tracking branch 'origin/stable-2.8'
Conflicts: lib/bdev.py lib/cmdlib.py
Signed-off-by: Michele Tartara <mtartara@google.com>Reviewed-by: Klaus Aehlig <aehlig@google.com>
Disk templates are "allowed" and not "enabled" in ipolicies
The way disk templates were displayed in instance policies became confusingsince the introduction of the enabled_disk_templates cluster parameter.
Fix issue 440.
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>...
Unit test for FillIPolicy keeping unknown keys
This makes sure that issue 401 doesn't come back.
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>Reviewed-by: Helga Velroyen <helgav@google.com>
Merge branch 'stable-2.8' into master
Add command generator for DRBD 8.4
The DRBD84CmdGenerator class, which generates commands suited for DRBD8.4, is added. A common baseclass for DRBD83CmdGenerator andDRBD84CmdGenerator is introduced as well.
Add test for version dispatching code
Based on the version of DRBD (as reported through /proc/drbd), the DRBD8class chooses helper objects which adapt for the differences betweenDRBD <= 8.3 and DRBD 8.4. This new test verifies that the proper objectsare created....
Extract DRBD related tests
As the DRBD code was extracted from bdev.py, the corresponding testsshould as well be extracted into their own file. That's what this patchperforms.
Signed-off-by: Thomas Thrainer <thomasth@google.com>Signed-off-by: Michele Tartara <mtartara@google.com>...
Extract command generation logic from DRBD8
In order to easily support the DRBD 8.4 command syntax, this patchextracts all command generation logic (i.e. where calls to `drbdsetup`are assembled) into a new class DRBD83CmdGenerator.
Add `drbdsetup show` parser for DRBD 8.4
Common functionality between the DRBD 8.3 and DRBD 8.4 parser has beenextracted into BaseShowInfo. A test which verifies the behaviour isincluded, but the DRBD84ShowInfo class is not yet used within the DRBD8class....
Extract DRBD info parsing into drbd_info.py
Parsing the status (/proc/drbd) and `drbdsetup show` output takes quitea bit of code, which has to be augmented by even more with the DRBD 8.4support. So extract all the related classes into their own file.
Extract DRBD8ShowInfo class
This class parses the `drbdsetup show` output and represents it in aneasily accessible format. It got extracted so that 1) the DRBD8 classcan focus more on DRBD logic rather than parsing and 2) it's easier toadapt the parser to new formats....
Add test for DRBD 8.4 /proc/drbd parsing
The current parser already supports the output of DRBD 8.4 /proc/drbd,so add tests which actually verify this behaviour.Note that the empty lines in proc_drbd84.txt are actually present in theoutput of DRBD 8.4, they always appear when minors are not contiguous....
Add test for empty version in /proc/drbd
The previous patch removed a (rather implicit) test for /proc/drbd notcontaining a version line. This adds an explicit test for this case.
Extract /proc/drbd parsing code into DRBD8Info
As the DRBD8 class got bigger due to the previous merge of BaseDRBD, nowparts of it are ripped out into DRBD8Info. This new class parses/proc/drbd and exposes the information in an easily accessible way. This...
Remove BaseDRBD
BaseDRBD was probably useful when DRBD 0.7 and DRBD 8 were supported.However, there is only one subclass of BaseDRBD remaining (DRBD8), andthe separation of responsibilities between those two classes was ratherrandomly chosen.
The unification into one class also eases the introduction of...
Rename _MassageProcData to _JoinProcDataPerMinor
That's what the method actually does, so there is no reason why weshould use a funny but unreadable name.
Unit test for cli.FormatPolicyInfo()
We try to parse the string obtained by formatting a policy, and see if weget the same policy.
Add command-line support for multiple specs in ipolicy
Command line options accept multiple min/max specifications in instancepolicies. The output of show-ispecs-cmd is updated accordingly.
Add multiple min/max specs in instance policy
Now instance policies can contain more than one min/max specs. This is themain element of the "Constrained instance sizes" section in the"Partitioned Ganeti" design doc.
This is a big patch, but changing the type of a configuration item requires...
Separate checks for std spec compliance
This is needed to be able to validate the std spec against multiple min/maxspec pairs (appearing in next patches).
Merge branch 'stable-2.7' into stable-2.8
Conflicts: NEWS doc/rapi.rst
Signed-off-by: Klaus Aehlig <aehlig@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Reason trail implementation for "start"
Opcode-specific implementation of the reason trail for the instancestartup operation.
Signed-off-by: Michele Tartara <mtartara@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Reason trail implementation for "shutdown"
Opcode-specific implementation of the reason trail for the instance shutdownoperation.
Add unit tests for cfgupgrade with a real configuration
Test upgrade and downgrade from a realistic 2.7 configuration.
Split functions in cfupgrade unit tests
This makes easier to test real configurations.
Limit specs in ipolicy command lines are always complete
Command line options are brought in line with the specs change of previouspatch. Old options are still allowed in gnt-cluster init, where thesemantic will remain non-ambiguous even after introducing multiple specs....
Refactor tests in ipolicy validation
This makes the code more modular, in preparation for the following patches.
Limit specs in instance policies are always complete
Specs used to specify limits in instance policies are always complete, evenat group level (only the whole limit set can be overridden). This is inpreparation for introducing multiple limits.
Add --ipolicy-xxx-specs options
These options allow to specify whole instance policy specs. This is neededfor the upcoming changes that tend to threat specs as monolithic objects.
Add command to print ipolicy options
The output of this command can be used to create an exact copy of thecurrent instance policy specs. The command could be expanded to print allthe options used to create a group or the cluster.
New CLI input type: list of key/value pairs
This will be used for the new instance specs options.
Allow "default" as a value for ipolicy templates
Previously the string value was being corrupted.
Extract DRBD-related classes into block/drbd.py
This is in preparation to introducing support for DRBD8.4.base.py had to be extracted as well in order to avoid cylic importsbetween bdev.py and drbd.py. It now contains the BlockDev class andutility functions needed by bdev.py and drbd.py....
Move lib/bdev.py to lib/block/bdev.py
That's in preparation of extracting DRBD related code from bdev.py. Asbdev.py is already rather long, new features will require to split itinto more manageable pieces. That's why it's moved in an own directory.
Reason trail implementation for instance reboot
This commits allows ganeti to correctly forward the reason trail informationregarding instance reboot.
Signed-off-by: Michele Tartara <mtartara@google.com>Reviewed-by: Helga Velroyen <helgav@google.com>
Remove old "reason" implementation
Remove the useless parts of the old, partial, implementation of the support fortracking the reason of instances state change, before implementing the newreason trail support, as per the design document.
Signed-off-by: Michele Tartara <mtartara@google.com>...
Update QA tests to check disk and NIC names
Modify existing tests to use the name option for instance disks. Theconfiguration options 'disk' and 'disk-growth' are merged into a single'disks' option, which is a list of dictionaries with 'size', 'growth' and...
Check that device names are unique and valid
Extend the CheckArguments phase of LUInstanceCreate and CheckPrereqphase of LUInstanceSetParams to also check if the name parameters ofdisks and NICs are unique and valid.
Signed-off-by: Christos Stavrakakis <cstavr@grnet.gr>...
Support UUIDs and names when refering to a device
Modify _ApplyContainerMods function to lookup NICs/Disks not only by theirindex inside the container, but also by their UUID or name. Abstract thelookup code in new GetItemFromContainer function.
Make type of identifier in "opcode._TestInstSetParamsModList" to be...
Move HooksMaster out of the mcpu module
We need to do this, so that backend.py doesn't need to import mcpu, andthus indirectly cmdlib. This reduces the size of the node daemon byabout half, which is very important as it is pinned in memory.
This solves Issue 419....
QA: Decision about disk templates to use is more modular
qa_config.IsTemplateSupported() is already used to decide which disktemplates are supported by QA. Enabled templates now are part of thatdecision.
Unit tests are updated.
Unit tests for config.ConfigWriter.VerifyConfig()
The test is aimed mostly at instance policies, which changed recently.
Refactor ispecs in ipolicy structures
Minimum and maximum instance specs are put together into a single elementof the instance policy. This is in preparation for introducing multiplemin/max specs.
cfgupgrade: Add --downgrade option
It's now possible to go back to the previous stable version. Unit testsprovided.
This is mostly useful during development, when going from master tostable/devel.
Unit test for cli.CreateIPolicyFromOpts()
Useful for subsequent patches.
Generic function to print data for gnt-xxx info
The function produces a human-readable output, which is also a valid YAMLfile, from an intermediate data structure. This will be used by "gnt-xxxinfo" commands. The output of "gnt-xxx info" was almost YAML-compliant, and...
Fix bug in rlib2 unit tests
The "queryargs" parameter of the init function of rlib2 classes shouldbe a dictionary, as defined in the ResourceBase parent class (inlib/rapi/baserlib.py).
In the rlib2 and baserlib unit tests, when the queryargs parameter is not...
Add the core of the monitoring daemon
This commit adds the core infrastructure of the monitoring daemon,and integrates it in the build and test systems.
The actual functionality of the monitoring daemon is still completelymissing.
Add infrastructure for allowing additional logfiles
Some daemons will need more than the single logfile that is currentlyallowed. This patch introduces the infrastructure to allow this.
Signed-off-by: Michele Tartara <mtartara@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Make gnt-node list -o(p|s)inst_list output stable
Currently, both the Python and Haskell code return the internalinstance list unsorted, which means the output can vary depending onthe phase of the moon (well, the Haskell code actually uses internallya tree, sorted by the instance name, but it's implementation detail)....
Merge branch 'devel-2.7'
Unit tests for objects.InstancePolicy + a fix
Tests for: objects.InstancePolicy.CheckParameterSyntax() objects.InstancePolicy.CheckDiskTemplates() objects.InstancePolicy.CheckISpecSyntax()
Instance policies with an empty disk-template list now are reported....
Unit tests for objects.FillIPolicy() + small fix
IPOLICY_DEFAULTS is now a legal policy (the disk-templates entry was a setinstead of a list, before).
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Fix upgrade of policy in objects.Cluster
Unknown elements were silently removed on startup. This means that asoftware upgrade could result in lost configuration information ifcfgupgrade wasn't run promptly.
Added unit test for Cluster.UpgradeConfig() to cover this case....
Fix instance policy checks for default back-end parameters
Policy violations of back-end parameters that used the cluster defaultvalue were not reported in cluster-verify.
Unit tests for cmdlib._GetUpdatedIPolicy()
Not 100% coverage, though.
Fix policy check for disk templates
Instance disk template is checked against the policy, and disklessinstances aren't checked for the number of disks.
Allow rpc.MakeLegacyNodeInfo to parse non-LVM results
'MakeLegacyNodeInfo' is not the best place for this, but we'd have toduplicate it if we wanted a LVM-less version, so the easiest is to addan optional parameter that allows it to accept/skip LVM-less results....
Test AsyncStreamServer with abstract unix sockets
This was meant to be since "the beginning" but was never submitted assomehow it failed with python 2.4. Now that the minimum python versionhas been increased it can be added.
Signed-off-by: Guido Trotter <ultrotter@google.com>...
qa_config: Support callable test checks
Until now all tests depended on simple AND/OR conditions. With theintroduction of virtual cluster support, some tests can only run ifvirtual clusters are not in use (e.g. not yet supported or notpossible). Instead of introducing more logic for enabling/disabling...
QA: Add config entries for virtual cluster
If these new options are set, a virtual cluster is created and used.Virtual clusters consist of 1..N virtual nodes on a single physical (orvirtual) node. The design of virtual clusters is described in“doc/design-virtual-clusters.rst”....
Fix style violations in commit 4a90bd4, small changes
- Commit 4a90bd4 contained a rather large number of style violations: - Indentation/sequence formatting - Wrapping of long lines - Docstrings whose first line was wrapped - A stray backslash in a docstring...
RAPI: Add flag to require authentication
Most RAPI resources do not require authentication for the “GET” method.In some setups it can be desirable to always require authentication.This patch adds a command line parameter to always require it.
Some unrelated minor typos in the “ganeti-rapi” man page are also fixed....
qa_config.AcquireNode: Use sort key function
Use a sort key function instead of using a comparing function (“cmp=…”).The latter is not supported in Python 3 and using a sort key function iseasier.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>
qa_config: Add repr for instance/node classes
Having a readable representation makes debugging easier.
Test "GetNodeInfo" for Xen hypervisor
This patch adds unit tests for success and failure of the “GetNodeInfo”method for the Xen hypervisor abstractions.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Helga Velroyen <helgav@google.com>
QA: Add validation for cluster name
baserlib: Make "_OPCODE_ATTRS" public
"_OPCODE_ATTRS" will also be used in the Sphinx extension.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Helper to retrieve access permissions for RAPI resource
Merge similar code into a helper function.
Status change reason support for Reboot
Add support to the Reboot command for specifying the reason for the laststatus change.
Some features are implemented as functions, even if used only once, becausethey will be used by the future patches introducing reason support for all...
Infrastructure for specifying instance status change reason
This patch introduces some infrastructural modifications that will be used bythe following commits to implement the support for specifying the reason forthe last status change of an instance....
Add IQ_NETWORKS in query tests
Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>Reviewed-by: Helga Velroyen <helgav@google.com>
Rename lib/objectutils to outils.py
Back when this was introduced, I mentioned that it breaks heavily tabcompletion (ob<TAB> doesn't work anymore), but at that moment I didn'thave a suggestion what to name it. I think outils is good and shortenough, and doesn't conflict with anything else, so here it goes....
Finish the remote→restricted commands rename
The documentation still points to /etc/ganeti/remote-commands,although the code is already using restricted-command. Update thedocumentation and a few docstrings accordingly.
Signed-off-by: Iustin Pop <iustin@google.com>...
Add unit tests for RADOSBLockDevice
Add unit tests for the RADOSBlockDevice JSON output parsing function,and modify the unit tests for the plain output parsing function towork with the new code.
Signed-off-by: Stratos Psomadakis <psomas@grnet.gr>Reviewed-by: Guido Trotter <ultrotter@google.com>
Moved uniformity check for exclusive_storage flag
Cluster-verify used to check that the value of exclusive_storage is uniformwithin node groups. Now, it's impossible to change the flag for a singlenode, so that check has been removed and an equivalent one has been added...
"exclusive_storage" cannot be changed on single nodes
There's never been support for a configuration where nodes in the same nodegroup have different values of the exclusive_storage flag. This patchdisables the possibility to change the flag for individual nodes....
Upgrades made on loading the configuration are always saved
Before, only some upgrades were written back to the configuration file. Alittle refactoring of _UpgradeConfig() has been done to write unit tests.
QA: Convert nodes to objects
Up until now nodes were stored as a dictionary. The keys were hardcodedin a lot of places and entries modified directly.
This patch introduces a new class for nodes in QA named “_QaNode”. Itstill supports accessing details via dictionary syntax, but that will be...
QA: Release instances through method
Instead of having a module-level function in “qa_config” to release aninstance, a new method on instance objects is used.
QA: Convert instances to objects
Up until now instances were stored as a dictionary. The keys werehardcoded in a lot of places and entries modified directly.
This patch introduces a new class for instances in QA named“_QaInstance”. It still supports accessing details via dictionary...
Refactor storage of runtime exclusive storage flag in QA
This is a follow-up for “qa_config: Remove exclusive storage flag fromconfig”. Instead of storing the flag in a module-level variable it isnow stored within the new QA configuration class and unit tests are...
Refactor QA configuration code
Ever since its introduction (sometime before commit cec9845 in September2007), the QA configuration was stored in a dictionary at module-levelin “qa/qa_config.py”. The configuration was loaded, verified andevaluated using module-level functions. Since then the configuration has...
vcluster: Make _MakeNodeRoot public
The QA code will also have to generate virtual paths and this functioncomes in handy.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Michele Tartara <mtartara@google.com>
Extract container converters
“objects.ConfigObject” contains two useful functions for working withcontainers of serialized objects, “_ContainerToDicts” and“_ContainerFromDicts”. This patch separates those functions and movesthem into “objectutils” as they'll be useful for converting parts of the...
objects: Improve handling of TCP/UDP port pool
- Handle de-serialization correctly when pool is not defined- Serialize to empty list when the attribute is None (this should never happen in reality as the attribute is always set when de-serializing)- Add tests...
Add a helper function for hypervisor verification
This will allow easier multi-error results from hypervisors; rightnow, we only report the first error, which is not nice.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>