Migrate lib/uidpool.py from constants to pathutils
File system paths moved from constants to pathutils.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Migrate lib/tools/ensure_dirs.py to pathutils
Migrate lib/ssconf.py from constants to pathutils
Migrate lib/ssh.py from constants to pathutils
Migrate lib/mcpu.py from constants to pathutils
Migrate lib/ovf.py from constants to pathutils
Migrate lib/rapi/baserlib.py from constants to pathutils
Migrate lib/rpc.py from constants to pathutils
Migrate lib/server/*.py from constants to pathutils
Migrate lib/hypervisor/*.py from constants to pathutils
Migrate lib/{jqueue,jstore}.py from constants to pathutils
Migrate lib/luxi.py from constants to pathutils
Migrate lib/masterd/instance.py from constants to pathutils
Migrate lib/client/*.py from constants to pathutils
Migrate lib/cmdlib.py from constants to pathutils
Migrate lib/confd/*.py from constants to pathutils
Migrate lib/config.py from constants to pathutils
Migrate lib/backend.py from constants to pathutils
Migrate lib/bootstrap.py from constants to pathutils
Migrate lib/cli.py from constants to pathutils
Migrate build-bash-completion from constants to pathutils
Migrate qa/*.py from constants to pathutils
constants: Move most paths to separate module
This is inpreparation for the implementation of virtual clusters. Manypaths will change based on an environment variable and are no longerconstant and should no longer be in “constants.py”. Since “constants.py”...
configure: Provide options to change default user/group
This is needed for virtual clusters where a non-root user is used to runthe daemons.
constants: Add missing word in comment
It was missing in commit 243cdbcc on May 7, 2008.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Agata Murawska <agatamurawska@google.com>
QA for instance migration made more compact
Also, the value of always_failover is not hard-coded any more, and it'spossible to disable changing the always_failover parameter, if needed.
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
Always_failover doesn't require --allow-failover anymore
If an administrator sets always_failover, it means that there is no needfor another explicit approval to failover instead of migrating.
Remove useless lock in recreate-disks
The old primary node was being locked to do RPC, but we don't lock nodesfor RPC anymore, so the lock is useless.
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Update design for virtual clusters
The biggest change is going from a command line parameter for specifyinga daemon's directory to using an environment variable. Some other thingsare clarified.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: René Nussbaumer <rn@google.com>
Small style adjustment
Signed-off-by: René Nussbaumer <rn@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
Make iallocator request types more strict
This adapts the type of some request input fields to more stricter ones.
Rework the required_nodes field
This was used to verify the result of instance allocation. However, thisis not logic which belongs to the cmdlib rather than to the iallocatorrequest itself. So abstract it by its own function. And update theValidateResult accordingly....
objects, rpc: Code cleanup
objects.Disk.ComputeLDParams: Simplify structure, don't use andoverwrite local variables.
rpc.AnnotateDiskParams: Replace explicit loop.
RAPI client: Fix typo in docstring
Also update the copyright header.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Disable pylint warnings for unknown members
For some reason pylint can't determine, that the members are calculatedupon runtime and complains. This is weird as the same mechanism works onopcode/objects. I haven't found the issue for it yet, so let's just...
Adapt gnt-debug iallocator
Add the new multi-alloc request to the test allocation framework. Fornow we just create --count identical instances. This might need furtherimprovements later.
Adding new multi-allocation request
Now that we've the refactoring done, we can easily add the new mode andeverything works.
Switch all iallocator calls to the new one
After the refactor, the old calls are not working anymore, so we switchto the new ones.
Refactor IAllocator code
The IAllocator class was handling all the requests on its own, passingin parameters on top level which works, but is hard to maintain and notflexible.
With the upcoming change to the IAllocator for MultiAllocate we can'tuse the toplevel parameters anymore. Therefore, we refactor the code...
Make the slots functionality more modular
As we will introduce another set of containers using the slots trickwe abstract away as much as possible to separate bases classes. Thechild classes then adapt them for their needs. This leads to less code...
Move _MakeLegacyNodeInfo to rpc
This function will also be needed for the iallocator refactor. Also thisfunction does not belong to cmdlib but rather to rpc. Lets move itthere.
Move _CalculateGroupIPolicy and _ComputeDiskSize
This makes it possible to better untangle the IAllocator code, whichwould otherwise lead to a recursive import structure as we need thosehelpers there as well.
Signed-off-by: René Nussbaumer <rn@google.com>...
Update documentation for "gnt-job list"
Commit f037e9d7 added command line flags to filter jobs, but thedocumentation was not updated. Also mention the ability to usemore complex filters.
Simplify a bit more the test harness
We can build the test groups directly in the `testSuite' helper,instead of doing it (much later) in the test harness.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Remove the slow/fast tests functionality
Since the recent commits improved the speed of the two "slow" testgroups to regular test speed, we can remove this kludge and simplifysignificantly our test runner, yay!
Signed-off-by: Iustin Pop <iustin@google.com>...
Fix arbitrary ConfigData object generation
The Cluster object, as it is defined right now, has many '[String]'members, which means that in a standard arbitrary generator these willbecome very big, which is the reason for the current slowness of thetest 'Config_serialisation'....
Improve the `AllocPolicy' test
This test has a few deficiencies, which this patch addresses:
- using arbitrary 1 or 2 node count for allocation is obsolete, nowadays we need to use a number appropriate for the instance's disk template (and we should remove that parameter…)...
Improve the `CanTieredAlloc' test
Currently, this test is very slow. Upon investigation, this is due tohow `tieredAlloc' works:
- tries to allocate one instance- if failed, shrink the instance by the "most failed" resource- restart
In this algorithm, if the "most failed" resource is e.g. memory, and...
Add new test for checking multi-allocations
This test expands the "single-alloc-no-rebalance" by allocating a fewinstances on a small cluster, and ensuring that after we allocate allof them, either we can't rebalance or if we rebalance the scoreimprovement is very small....
Improve the prop_Alloc_sane test to detect mis-allocations
Currently, this just checks that a cluster cannot be rebalanced aftera single instance allocation. However, we can also test whether theallocation decision computed a correct new cluster score, by checking...
Merge branch 'devel-2.6' into submit
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Agata Murawska <agatamurawska@google.com>
Merge branch 'stable-2.6' into devel-2.6
Fix bug in non-mirrored instance allocation
The function `allocateOnSingle' has a bug in the calculation of thecluster score used for deciding which of the many target nodes to usein placing the instance: it uses the original node list for the scorecalculation....
Further hlint fixes
Commit 2cdaf22, “Re-enable standard hlint warnings”, got it almostright. The only problem is that (confusingly) the default set of hintsis not in HLint.Default, but in HLint.HLint (it includes Default andsome built-ins).
After changing the lint file to correctly include the defaults, we had...
Rework CLI modules and tests
While investigating how we could test the Daemon.hs module, I realisedthat we have a very, erm, sub-optimal situation:
- HTools/CLI.hs has a nice IO/pure separation testing in cmdline parsing, which allows some basic functionality to be tested, but...
Fix deserialisation bug in ResultEntry
Found via the newly added unit-tests, which test most of theserialisation code in Query/Language (except for QueryResult, forwhich we already tests both sub-components separately).
Add query filter tests
These tests are node specific only because we don't have other querytypes implemented yet, but what they actually test is the variousfilter types.
The tests are trying to cover most filter functionality; missing fornow is proper checking for ContainsFilter and TrueFilter, the rest...
Add some unittests for node queries
These new tests check that:
- no known fields return unknown- any unknown field returns unknown- the type of the fields is consistent between the getters and the field definition- the length of each result row corresponds with the number of fields...
Add a small 'passTest' helper
This is symmetric to failTest, and allows us to use it in cases wherewe need to return a property.
While replacing 'property True' with it, I saw a case where we cansimplify the use and thus reworked that check.
Add entire ConfigData serialisation tests
Using the recently-added genArbitrary, we can now implement Arbitraryinstances for even "huge" objects like Cluster, so let's use that toimplement entire ConfigData serialisation tests.
Note that, as we don't have yet proper types for some of the Params...
Replace manual arbitrary instances with genArbitrary
There are a few more that could be replaces, once we start usingappropriate (new)types.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: René Nussbaumer <rn@google.com>
Introduce a helper for simple Arbitrary instances
After getting really annoyed at yet another "<*> arbitrary", I thoughtthat we should be able to automate this, at least for types which aresimple enough and have already all the "prerequisites".
Hence the new genArbitrary function and its helpers, which can:...
Add Instance serialisations tests
This is not perfect, as for many of the parameters we don't have goodArbitrary instances, but is better than nothing.
Add a test helper for simple JSON serialisation testing
While adding yet another JSON serialisation test, I realised that thiscan be trivially abstracted; hence this patch, replacing both simpleversions (readJSON . showJSON == id) and the standard version (with...
Add unit test for serialisation of DiskLogicalId and Nodes
Since the DiskLogicalId type is manually serialised/deserialised (seeObjects.hs, `encodeDLid' and `decodeDLId'), let's add a test thatchecks that these are idempotent when combined.
Since we're at it, let's add the same test for Node serialisation,...
Add unittests for the BasicTypes module
This adds test properties for the various laws that the instances ofResult should follow; I could not find (offline) laws about `mappend',but otherwise I implemented all laws that I could find.
Note that we have to silence hlint warnings for the things we want to...
Fixup test suite names
The names were not in a proper hierarchy, leading to inconsistenciesabout what they were actually tested.
We change this by reproducing in the test names the relative hierarchywithin the Ganeti directory, leading to nicer test suite names (in...
Simplify property and test case names
Since we now have separate namespaces due to the multi-file split, wedon't need to keep the name of the module in the property names, as wedon't have so many potential conflicts anymore.
We remove the group prefix handling from TestHelper and simply do a...
Move Version.hs up from under HTools/
This is another module that is generic, and not htools-specific.
Two tiny makefile fixes
During the recent moves and renames, two things have slipped through,since I didn't run make check-local…
Remove QC.hs and replace it with an auto-generated file
In order to have correct code coverage results, we must somehow importall production modules into the test runner. Until now, this was donemanually (when we didn't forget) in QC.hs.
To improve the situation, we remove QC.hs and replace it with an...
Split the Makefile variable HS_LIB_SRCS
This currently holds both production files and test files; since wewant to treat these separately, let's split the variable in two, andintroduce another one for the old, inclusive one.
Split last two remaining tests from QC.hs
This makes QC.hs obsolete, but we will remove it in a later patch,once we automatically build a file with all the imports (to ensurecorrect coverage results).
Move JSON.hs and Compat.hs out from under HTools/
These two files are not htools-specific, so let's move them out of theHTools subdirectory/module hierarchy and directly under Ganeti.
Split most HTools test code into separate files
Except for Ganeti.HTools.JSON, which needs rename, we split all theother test suites into separate files.
We have to add another common test helper, due to import dependencies(sigh), but otherwise this split is pretty straightforward....
Split Luxi, Qlang, Ssconf and OpCodes tests
… from QC.hs into their own files, again mirroring the production codesource tree.
Split Rpc tests from QC
This required lots of other code moves, so I created it as astandalone patch.
Split one more module out of QC and add test helpers
This splits the confd/utils tests, and adds the TestCommon module forshared test code.
Split out Objects.hs from QC.hs
This is the first file split out from QC.hs - an easy one, since ithas just one test.
The patch changes the way we build hpc excludes, since now we'll havemany modules that need to be excluded, and hpc doesn't seem to be able...
Move the unittest helper to a new Test/Ganeti dir
This patch starts the move of the test haskell code from`htools/Ganeti/HTools/' to its more proper place of`htest/Test/Ganeti'.
Move generated htools test stubs to htest/
This moves the last (I think) htools-related bits out of test/ underhtest/.
Move shelltests from test/ to htest/shelltests/
This is another rather trivial patch, moving all the (htools)shelltests to their own directory.
Move htools test files from test/data/htools to htest/data
Following up on the program moves, we now move the test data files.
Move haskell test code to htest/
This is the first commit of a series that will attempt to cleanup thetest code organisation, which evolved somewhat organically from theinitial pure htools functionality.
The proposed organisation of the tree will be as follows:...
Fix gnt-debug iallocator
There was an issue with the recent ipolicy introduction which lead to abug in gnt-debug iallocator. It was not providing the spindle_use fieldand therefore it wont let you create a valid iallocator request.
Implement compilation of regexes at creation time
This means that the verification of the correctness of the regex isdone once, at the deserialisation/creation time, as in the Pythoncode. To do this, we have to change the FilterRegex type from an alias...
Rename the Qlang and Queryd modules
Per the new query module hierarchy, rename Qlang to Query/Language andQueryd to Query/Server. This way, all query-related functionality isnow "contained" in the Query/ directory.
Add filtering support in Query
This adds basic infrastructure for filtering (fully functional except,as usual, for runtime data), and then uses it for node queries.
Since the filtering exports regex matching as an externalfunctionality, we have to use a regex library. There are many flavours...
Add an Applicative instance for our Result type
This will allow us to run generic computations in an Applicativecontext, if a monadic one is not needed (or not applicable due toclass constraints).
Parameterize the Filter type
In preparation for introducing filtering functionality, we convert the'Filter' type from a '*' kind to a '* -> *' kind.
This allows us to define some general properties for the filter, andfor example introduce later an easy filter compilation, etc....
Implement QueryFields for Nodes
Since we have all the definitions already, we can easily enablethis. Manual testing shows no difference between the Python and theHaskell versions of node list-fields.
Merge branch 'devel-2.6'
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>
Putting the multiallocate pieces together
This is the final part:
Signed-off-by: René Nussbaumer <rn@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Adding allocList function
This function iterates over the tryMGAlloc function, updates the nodelist and instance list, and refeeds it recursively until no newinstances are left.
This allows us then to allocate multiple instances and see if they wouldfit together....
Adding new IAllocator mode to constants
Fix warnings/errors with newer pylint
To help developing Ganeti on newer distributions, let's try to fixpylint warnings/errors. I'm using pylint from current Debian wheezy:pylint 0.25.1, astng 0.23.1, common 0.58.0, and we have 3 things thatneeds fixing....
Signed-off-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Fix decorator uses which crash newer pylint
Pylint version:
pylint 0.25.1, astng 0.23.1, common 0.58.0
crashes when passing the fully-qualified decorator name with:
File "/usr/lib/pymodules/python2.7/pylint/checkers/base.py", line 161, in visit_function...
Instance autorepair design
This design describes a tool that will perform automatic repairs oninstances when they are detected to be unhealthy (living on offline ordrained nodes, at the moment). These repairs can be scheduledautomatically or requested as a one-off by a tool or person....