CLI: Prevent incompatible options to be selected
This patch makes CLI abort if more than one backend is selected.
Add support for luxi backend in CLI/hspace/hbal
This patch changes the backend selection method in CLI to prefer, in order: - a RAPI specification - a Luxi specification - and finally the node/instance files
It also modifies hspace and hbal to provide a ‘-L’ command line option...
Initial commit of the luxi backend
This patch adds a luxi backend that allows direct query of the masterdaemon on the local node. This patch doesn't enable the backend to beused.
There are a couple of things still missing in the implementation: - we don't have a master timeout in reads and writes, only a...
Introduce timeout in RAPI queries
The patch adds two constants in Types.hs for connect and query timeout,then modifies Rapi.hs to use them as the connect and general curltimeout.
Rapi could be improved more, as currently we wait double the totaltimeout due to not aborting early in case the node queries failed.
Fix a haddoc issue
hspace: fix failure handling of tryAlloc results
Currently hspace doesn't handle failures from tryAlloc correctly; thispatch changes the iterateDepth function in hspace to return a Result (…)so that errors can be propagated correctly.
The patch also changes one output key to be more clear and a typo in...
Change the tryAlloc/tryReloc workflow
Currently, the tryAlloc and tryReloc function return a list with all theresults, both failures and successes. This is fine for hail, which doesone round of allocations, but is not so good for hspace, which doesiterative rounds; since at each (successful) step we only take the best...
Simplify the Cluster.tryAlloc structures
Currently the tryAlloc function calls theallocateOnSingle/allocateOnPair and the builds a new tuple with thosefunctions's result plus the new node list. This is however suboptimalin two respects: - the new nodes added are the 'old' versions of the respective nodes,...
Slight change to the internal allocation results
Currently the Cluster.AllocSolution type is defined as a list of‘(OpResult Node.list, …)’ and the results for applyMove are defined as‘(OpResult Node.List, …)’. Both these means that the failure/successindication is hidden in the first elements of this tuple, which makes is...
hspace: switch output to shell-script format
This (big) patch changes the output of hspace from text-format(separated by ‘: ’) to a shell-snippet, in ‘key=value’ format.
This will allow sourcing the output or parsing it via awk/sed/etc.
hspace: move instance count and score into CStats
Currently the instance count and cluster score are separated from theother initial/final phase stats, even though they are very similar. Thispatch moves computation of these two into totalResources/CStats and...
Fix unittests
The recent OpResult and CPU values additions broke unittests.
Export more stats in hspace
This patch changes Cluster.totalResources to compute more resources andprints them in hspace.
Show errors on stderr instead of stdout
Currently many of the exit and warning conditions mistakenly display errormessages on stdout, which makes parsing the output of programs harder. Thispatch attempts to fix such occurrences.
Fix score calculation to work with empty clusters
Currently the cluster score calculation includes an offline instancepercentage, expressed as “offline inst / (offline + online inst)”, whichresults in NaN for empty clusters. This patch changes the calculation...
Optimize the Utils.stdDev function
This patch optimizes the stdDev function in two respects: - first, we don't do sum . map which builds an intermediate list, but instead use a fold over the list to build incrementally the sum; this should reduce both the time and space characteristics, as we...
Take the foldl out of Loader.fixNodes
Currently Loader.fixNodes is foldl' with a complicated function. Itmakes more sense to take foldl' out of this function (and put it intothe caller) and let fixNodes be only this internal function.
Simplify Cluster.computeMoves
This patch changes the function Cluster.computeMoves to use guards and acouple of subexpressions in order to greatly simplify it.
Fix hlint-generated warnings
This big patch cleans up the code per hlint indications. Many removalsof extra parentheses, replacements of concat . map with concabtMap,extra dollar signs, eta reductions, etc. were performed.
The code still compiles and passes a couple of manual tests on sample...
Add computation of the failure reason in hspace
This patch enhances hspace to report why the allocation sequencestopped, both in absolute error count and for the top reason.
Return correct failure data from Node.add*
This patch alters the Node.addPri/addSec to return correct failure data.It removes the computeFailN1 function from the module as that used tocombine both mem and disk checks in the same function and thus the real...
Introduce a new type for allocation results
Currently the allocation/move operations workflow return ‘Maybe a’,which is very convenient but loses all details about the failure mode.
This patch introduces a new data type which encodes the specific failure...
Remove hn1 and related code
hn1 was deprecated for a while and this patch removes it altogether. Thesupport code in Cluster.hs is also removed.
Fix totalResources avail disk computation
This uses the newly-added Node.availDisk to compute the actual availabledisk correctl, and display the total allocatable disk in hspace.
Add an availDisk node function
This function returns the amount of available disk, which depends onwhether a low disk limit has been configured or not and on the free diskspace of the node.
Add two new autocomputed vars to Nodes
Currently we track the max disk usage/max vcpus as percentages, howeversometimes it's easier to check against minimum free disk or maximumnumber of cpus, as units instead of percentages.
This patch adds two new variables, lo_dsk, hi_cpu, which are recomputed...
Add a new type for cluster statistics
Currently totalResources returns a 5-tuple of integers. This is not easyto handle, as each change on the return type means that each caller mustbe updated.
This patch adds a new type for cluster stats and uses that instead as...
Add display of more stats in hspace
This patch changes Cluster.totalResources to compute more details aboutthe cluster status, and enhances hspace to display more of these.
Fix a haddock/docstring issue
Implement cpu/disk limits in instance moves
We modify Node.addPri/addSec to take into account the limits on instanceadds.
Add two new node attributes
Two new min disk free ratio and max cpu usage attributes are added to thenodes. These will be used in the future to restrict allocation.
Fix 'unused X' warnings
This removes some unused functions and imports to cleanup the warnings.
Fix the various monomorphism warning
In a few places (e.g. tryRead or any printf call) it's a little bit hardto add the correct type signatures, but in the it is possible to fixthese warnings (which can bite one in subtle cases).
Small changes to the node list output
This is just some cleanup of the node list output, adding pcpu/vcpucounters, and making the display slightly nicer.
Add cpu ratio to cluster calculation
Update cpu counters correctly after pinst changes
The cpu counters are update on primary instance adds/removes.
Add cpu-count-related attributes to nodes
This patch adds cpu-count related attributes to nodes: - total cpus - cpus in use - ratio of virtual:physical cpus
We also set correctly the cpu values at load time, but we don't doanything yet while moving instances around. The cpu ratio is shown in...
Add a new vcpus attribute to instances
This patch adds reading of vcpu count for instances, in preparation forusing the vcpu ratio in cluster scoring.
Fix reading of total disk space in iallocator
IAllocator currently uses a wrong key name for reading the total diskspace (‘disk_usage’ which was copied from RAPI, but the actualiallocator key is ‘disk_space_total’).
This patch fixes that and also makes iallocator always use this key,...
Fix the ReplacePrimary instance move
During a replace-primary instance move, on the real cluster the instanceis temporarily started on the secondary, and as such we must check thatthe secondary node can hold it for this duration. Currently the codedoes not, and depending on cluster scoring it will put instances on such...
Rework the tryAlloc/tryReloc functions
Currently tryAlloc/tryReloc do not return the new instance, as this isnot needed for IAllocator alloc/reloc requests. However, for computingthe space, the new instance is useful, so we modify these functions toreturn this information too....
Add an utility function for triples
Small doc change
And an alignment issue.
Ensure consistent naming of the tools
This patch makes sure that all references to the name of the software isganeti-htools, not simply htools.
Add copyright/license information
This doc-patch adds copyright and license information to (hopefully) allneeded files.
tests: move the test declaration in QC.hs
This patch moves the test declaration into QC.hs, so that test.hs has tobe modified only when we add a new test category.
Small whitespace change
Move some alloc functions from hail into Cluster
These are generic enough to be used from multiple places, they belongbetter in Cluster.hs than in the hail source.
Move the RqType and Request types to Loader.hs
These two will be more generic than now, and belong somewhere else -Loader.hs is a generic module for data loading, thus we move them there.
Cleanup an old function
Also replace a type with its synonim.
Lots of documentation updates
This patch does only doc build changes, doc changes and function movearound (for more logical documentation). It should have no impact at allon the code.
A simple test for Container.addTwo
Add some very trivial Instance tests
This is more of an exercise in QuickCheck than strong testing.
Finish removal of unused params from PeerMap
This completes the removal started earlier byt removeing the need topass the number of nodes to Node.buildPeers, which is now unused.
Add test infrastructure and initial tests
This patch adds a QuickCheck-based test infrastructure and initial testsbased on it. The PeerMap module has a 100% coverage ☺
Side-note: one has to read the source of QuickCheck to see how to use it(especially the Batch submodule), the docs are not enough…
Some cleanup of the PeerMap module
This patch removes some unused functions and does some cleanup of theremaining ones.
Remove unused parameters from PeerMap creation
We remove some unused arguments (added way back for compatibility withArrays, which we didn't use in the end). This makes the code clearer(and doesn't need the Ndx type to be an instance of Num).
Remove an unused type synonim
Add a separate function for looking up instances
Currently we (wrongly) use lookupNode to lookup instances, just becausethe name assoc list has the same type. This patch adds a separatefunction for it.
Add type synonyms for the node/instance indices
This is a first step towards full datatype renaming. That requires morechanges, so at first we only want to document clearly what is a nodeindex, what is an instance index, and what is a plain Int.
Change the module import hierarchy
This patch makes the Types module a base module, and Node/Instance onesimport it, from the previous (opposite) situation. This will allow inthe future to use newtypes for the index and name types.
Port offline node fixes from Rapi to IAllocator
The IAllocator source was copied from Rapi before the offline node fixeswere made. This changes such that offline nodes are accepted correctly.
Fix loading of plain instances via iallocator
Currently iallocator is broken when reading single-node instances (andwith an ugly error message). This patch fixes this case, by marking themwith secondary node “noSecondary” like the rest of the code.
Fix some haddock issues
Slash is a reserved char. Slash is a reserved char. Slash is a…
hail: do not allocate on offline/drained nodes
This patch implements filtering out of the offline/drained nodes andfixes a bug in IAllocator.hs parsing (similar to an older bug in Rapi.hsfrom where the code was copied).
hail: Implement non-mirrored instance allocation
This patch implements non-mirrored instance allocation, by allocating assecondary node “noSecondary”.
Implement hail allocate (for 2-node requests)
This patch implements allocate for two node requests. One node requestscan be done as soon as we have a valid allocateOn function for singlenodes.
Working implementation if relocate
This patch completes the implementation of hail relocate. It maps allvalid destination nodes through a ReplaceSecondary IMove, filters outthe failed relocations, computes the resulting scores and picks thelowest one.
Start implementing the hail functionality
This patch implements a very stupid (and broken) version of hail‘allocate’.
Remove ktn/kti from first half of loader
This patch removes the ktn/kti lists from most parts of the first halfof the loading sequence. Some remain as the [(String, Int)] is thenicest way to lookup names and get indices back.
Remove the ktn/kti from second half of loading
This removes the return of ktn/kti from Loader.mergeData and associatedfunctions.
Remove most uses of ktn/kti
This patch removes all uses of ktn/kti from the past-loader stages.
Add some utility functions for kt deprecation
These will be used to remove even more uses of ktn/kti in non-criticalpaths.
Remove some extraneous uses of ktn/kti
Since we have Node/Instance.name, we can now simplify a few constructs.
Strip the suffix from the names in the objects
This strips the suffix from the objects themselves, not only from thektn/kti vars.
Make IAlloc.loadData return maps
This patch makes the format of IAlloc.loadData be similar the same asLoader.mergeData.
Move common loading sequence in CLI
This patch moves the common loading sequence to CLI, such that hbal/hn1and possible future scripts that take the input from same sources canuse it.
Move checkData from Cluster to Loader
This moves the remaining loading function to Loader (together with itsassociated support functions).
Introduce a class for CLI options
This patch generalizes a little the CLI handling by not passing in aspecial function for help and such, but instead requiring that theoptions object supports some common functionality.
Add a small class for Nodes and Instances
Since both nodes and instances support some common functionality (namesand indices), we add a class so that we can access these attributes in ageneric way.
Add back names to nodes/instances
In order to simplify the data structures, we add back the name on thenode and instance objects. We still keep the index for, well, indexing,but we will use the name directly from the object, in order to get ridof the ktn/kti arguments which are passed around everywhere.
More code reorganizations
This new big patch does a couple of more cleanups in the loading of datachapter: - introduce a Types module that holds most types (except the base Node/Instance/etc.) so that multiple other modules can use these (instead of only Cluster and its users)...
Rework the loader model
This big patch changes the loader model from “string data as commonformat” to actual object structures as common format.
The text loading function move from Cluster.hs to a new Text.hs module,some common functions are moved to a new Loader.hs module, and the...
Initial work on hail
This makes hail compile and get a request parsed via IAlloc, but nothingmore.
Experimental support for non-redundant instances
This patch adds experimental support to hbal for non-redundant instances(i.e. instances with only one node). They are currently handled asnon-moveable, and as such the algorithm simply ignores them.
Supports needs to be added when reading from RAPI via hscan, and...
Small doc addition
Introduce nice errors on invalid input fields
This patch switches from plain read to a wrapper over readsPrec thatreturns better error messages than the buildin 'Prelude: no parse'.
Split node/instance parsing into functions
This allows easy checking for valid format of the input data (row-wise).
Add initial validation checks in Cluster.loadData
This patch converts loadTabular and loadData to a monadic form, thusallowing meaningful error messages from the node/instance load routines.
Small syntax improvement
D'oh, one can extract from a wrapped variable, not only from functions.
Convert Cluster.loadData to Result return
This patch changes Cluster.loadData to return a Result, instead ofdirectly the values; this will allow us to return meaningful errorvalues (e.g. when an instances lives on unknown node) rather than simplyabort. Currently the result is always an Ok, the actual signalling of...
Allow overriding the RAPI port/scheme
This patch adds a very dumb way to override the port and scheme forRAPI: if the master string doesn't contain a colon, we consider it asimple hostname specification, and thus prepend ‘https://’ and append‘:5080’, (the default RAPI port); otherwise, we consider it a fully...
Remove some 1.2 specific code
Now that we dropped 1.2 compatibility, we can removed the special casingfor RAPI differences.
Generalize some Result function into monad ones
We don't really needed, but is more clean like this.
Switch from hand-written monads to a real one
This big patch converts from our home-grown monad-like constructs(the Either stuff) to a real, Either-like-but-another-name monad.
We introduce a “Result a” monad, and this allows dropping many of theextra constructs. Hopefully the code is also more clear....
Small changes to the IAlloc module
Adding a small request type data structure.
Don't consider offline nodes as N+1 failed
This is just a cosmetic (I hope) change; the nodes shouldn't be usedanyway, and we only correct the display message.
Drop RAPI v1 compatiblity
Since we only support Ganeti 2.0, we drop the 1.2 compatibility.
Add support for 'offline' nodes
This patch drops compatiblity with Ganeti 1.2 and adds support foroffline nodes in the cluster. When reading from RAPI, the drained nodesare considered offline so that we don't allocate on them too.
Update the IAlloc module
We know have a working parseData function that returns the node andinstance data. This uncovered bad support for non-drbd instances ☹