History | View | Annotate | Download (78.1 kB)
Utility function for modifying an IORef using a lens
.. and a supplied function that works inside the lens.
Signed-off-by: Petr Pudlak <pudlak@google.com>Reviewed-by: Klaus Aehlig <aehlig@google.com>
New module for temporary reservation of config. resources
This patch adds the first step, the reservation of DRBD minors.
Merge branch 'stable-2.11' into stable-2.12
Merge branch 'stable-2.10' into stable-2.11
Merge branch 'stable-2.9' into stable-2.10
Conflicts: configure.ac # Taken both contributions
Signed-off-by: Hrvoje Ribicic <riba@google.com>Reviewed-by: Klaus Aehlig <aehlig@google.com>
Add an example for node evacuation
The configuration shows an unbalanced cluster witha node being drained. The natural evacuation strategyincludes frf-moves.
Signed-off-by: Klaus Aehlig <aehlig@google.com>Reviewed-by: Hrvoje Ribicic <riba@google.com>
Add configure option to pass GHC flags
Adding the HEXTRA option to make might not be practical for a changethat should be always applied, e.g., hiding a certain package. Thispatch allows the flags to be specified at the configure level.
Signed-off-by: Hrvoje Ribicic <riba@google.com>...
Implement getDisks in Confd
Add 'ReqInstanceDisks' request type and allow Confd to query for thedisks of an instance. The helper function 'getInstanceDisks' returns thelist of instances on the given node along with their disks and is usedby the function 'addInstNameToLv'....
Add systemd unittests
Test the consistency between daemon-util and the systemd unit files:
- Each service started/stopped by daemon-util must have a respective unit file. - The daemon user/group in the service file must be the same as the one reported by daemon-util....
Add systemd unit files
We add templates for systemd service files and systemd targets underdoc/examples/systemd and flag them to be built. The generated files arealso added to .gitignore.
Signed-off-by: Apollon Oikonomopoulos <apoikos@gmail.com>Reviewed-by: Jose A. Lopes <jabolopes@google.com>
Add a module with utility functions for MonadPlus operations
In particular, functions for retrying a MonadPlus action:It is repeated until it returns a valid result.
Conflicts: Makefile.am doc/design-draft.rst lib/bootstrap.pyResolution:...
Add design-node-security.rst to docinput
Currently, design-node-security.rst is not included in the dist tarball,making it impossible to rebuild the documentation using only the tarball.
Signed-off-by: Apollon Oikonomopoulos <apoikos@gmail.com>Reviewed-by: Helga Velroyen <helgav@google.com>
Add design doc for performance tests
This design doc describes which tests are added in order to test theperformance of Ganeti, specifically when handling multiple jobs inparallel.
Note that this design doc is submitted to stable-2.10 so performancechanges over different Ganeti versions can be captured....
Add basic test for CPU speed
In this example, the cluster has two nodes and four instances, two withprimary on each of the nodes. The most scarce resource on this clusterare (virtual) CPUs and the second node has 3 times the CPU speed of thefirst one. So distributing the instances 1 and 3 gives a more balanced...
Add lenses for OpCodes
...to simplify manipulation of them.
Signed-off-by: Klaus Aehlig <aehlig@google.com>Reviewed-by: Petr Pudlak <pudlak@google.com>
Add lenses for the job queue objects
...so that manipulations deep within such an objectget more simple.
Move the definition of JQueue objects to a separate file
Move all the definition of objects to a spearate file. Inthis way, the lense module for JQueue can use these objects,while JQueue can use the lenses. For use outside, we reexportthe objects.
Signed-off-by: Klaus Aehlig <aehlig@google.com>...
Add Haskell and Python modules for running jobs as processes
They will be used by Luxi daemon to spawn jobs as separate processes.
The communication protocol between the Luxi daemon and a spawned processis described in the documentation of module Ganeti.Query.Exec....
Add a module for livelock related functions
Currently it exports a function for creating livelock files.
Expose the configured Python path in Haskell
This is needed for properly executing Python job processes.
Create a Python submodule for jqueue
.. so that we can add new code into separate files, instead of adding itto jqueue.py, which has already grown too large.
Add a design to use CPU-speed as a resource
This design proposes to add relative CPU speed as aresource to be used in the Ganeti resource model.Implementing this design will solve issue 539.
Signed-off-by: Klaus Aehlig <aehlig@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Add a RunWithLocks QA utility function
This patch adds a QA utility function that acquires a set of locks, andattempts to run a given function with the locks in place. Should thegiven function block, this function does not detect this - laterpatches will address the issue....
Add 'xen-ifup-os' NIC script
Signed-off-by: Jose A. Lopes <jabolopes@google.com>Reviewed-by: Hrvoje Ribicic <riba@google.com>
Allow use of "and"
It is not necessarily bad coding stile to use "and" to presenta list of conditions. In fact, a statement like
and [ foo foo' , bar bar' , baz baz' ]
can be more readable than the equivalent
(foo foo')...
Verify locks can't be updated while having a pending request
Add a verify first sanity check for lock waiting: a user that has a pendingrequest cannot modify his locks. This patch also brings in the necessaryinfra structure for having arbitrary waiting structures....
Add data structure for lock waiting
To avoid jobs polling on locks, and also to ensure that themost important of the waiting jobs obtains a lock, add adata structure to keep track of who is waiting for which locks.
Move metad configuration related code to a separate module
... as an intermediate step before moving the responsbility ofcomputing the final OS parameter configuration from the configurationserver to the web server. This will allow the metadata daemon to...
KVM: move tap control functions to a submodule
Move all tap-related functionality to the hv_kvm.netdev submodule. Werename _OpenTap to OpenTap, since it will now be used as a publicfunction.
Also, change the hv_kvm tests to import the new code.
Signed-off-by: Apollon Oikonomopoulos <apoikos@gmail.com>...
KVM: move monitor & QMP classes to a submodule
We move the HMP and QMP classes to the hv_kvm.monitor module.
Signed-off-by: Apollon Oikonomopoulos <apoikos@gmail.com>Reviewed-by: Hrvoje Ribicic <riba@google.com>
Convert hv_kvm to a package
Weighing in at 3k LOC, it is probably time to split hv_kvm into a numberof modules before adding new code.
Add a WConfd RPC Client
The client is generated automatically by Ganeti.THH.HsRPC.
It also includes helper functions for creating a client object.
Add a module for creating Haskell-to-Haskell RPC clients
It uses TH to create client functions with the corresponding signaturefrom a given list of server functions.
Add metadata daemon server entry point
... which starts the configuration and web servers.
Add metadata daemon configuration server
The configuration server listens on a Unix socket for connections fromthe node daemon. The node daemon sends the instance parameters to theconfiguration server so they can be served through the metadata daemonweb server to the instances which have the communication mechanism...
Add metadata daemon types
The metadata daemon will be responsible for receiving the instanceparameters from the node daemon and keeping them in memory. The typeof the instance parameters in defined in a separate module because itwill be shared between the web server and the configuration server....
Move metadata daemon to its own directory
Currently, the metadata daemon is mainly a web server. However, a newconfiguration server will be introduced along with other metadatarelated modules. Therefore, this patch moves the current metadatadaemon web server to its own directory. This module is also renamed...
Merge branch 'stable-2.11' into master
Merge branch 'stable-2.8' into stable-2.9
Include qa/patch in Makefile
Fix the build by including the qa/patch directory in Makefile.am.
Signed-off-by: Thomas Thrainer <thomasth@google.com>Reviewed-by: Helga Velroyen <helgav@google.com>
Move message formatting functions to separate file
The formatting functions in qa_utils.py cannot be used by modulesimported there, such as qa_config. This patch factors the functioncalls into a separate file. Also reorders imports in touched files.
Add a mock for the wconfd module
As our so-called unit tests for the logical units are morelike integration tests, calling the actual LU through an almostlive mcpu, we need a quite involved "mock" for the wconfd module,called indirectly through mcpu. The difficulties arise from the...
Add a mock version of livelocks
...as the will be used by mcpu for lock handling and that willrequire tests to be adapted accordingly.
Add a module for constructing Ssconf from the configuration
This migrates the functionality from PythonConfigWriter._UnlockedGetSsconfValues to Haskell.
Add a module for handling compression
Since the Haskell 'zlib' library by default uses exceptions forsignalling errors, we need to use its internals to implement a proper,pure decompressing function.
Add a module for generated lenses for Ganeti.Objects
The module also adds type classes for lenses for common fields (serial,mTime, uuid and tags).
Splitting the lenses into a separate module avoids problems with unusedfunctions.
Signed-off-by: Petr Pudlak <pudlak@google.com>...
Add a module for custom generic Lens functions
The module currently re-exports Control.Lens and adds a function thatcreates lenses from data types by appending 'L' to its field names.
All Ganeti modules should import this module instead of Control.Lens...
Move helpers for specific fields into a separate module
These helpers aren't needed by other code in Ganeti.THH and since themodule is already large, it'll be better to separate them into theirown.
No functionality is changed by this patch, just functions moved....
Add utilities for liveliness lock files
To request resources from WConfD, requesters have to providethe name of a file they own an exclusive lock on. In this way,their death can be detected. Add utility functions to obtainsuch a file name.
Add a function to detect death of a job
With jobs running in processes different from the lock management soon,we need to detect if a job holding locks dies, in order to release theresources.
Add serialization for LockRequest GanetiLocks
Add a module describing how abstract requests, like lock requests arepassed over the wire to WConfD.
Fix the dependencies of Makefile.ghc
Include only the programs that were configured to be built, and let GHCfigure out the required Haskell sources by itself.
Signed-off-by: Petr Pudlak <pudlak@google.com>Reviewed-by: Helga Velroyen <helgav@google.com>
Include mond and confd only when configured
Previously they were included no matter of what the configuration was,failing to compile if some of the dependencies were missing.
Verify serialization/deserialization consistency for Locks
As we made Ganeti.Locking.Locks an instance of JSON, we better verifythat we can readJSON what we showJSON.
Provide fields for lock queries
For luxid to be feature-complete with respect to masterd, italso needs to answer requests about locks. This includes knowingthe fields available for locks.
Signed-off-by: Klaus Aehlig <aehlig@google.com>Reviewed-by: Petr Pudlak <pudlak@google.com>...
Add a module for functions specific to config. management
.. and include ConfigData in WConfd's ConfigState.
Currently it includes functions for asynchronous saving theconfiguration and reading and (over)writing it in the daemon state.
Load the configuration from the file at daemon startup and configure an...
Add a module for creating and using asynchronous workers
Each such worker has its own background thread and performs a giventask when triggered. Triggers arriving when the worker is processing areaccumulated and processed together in a subsequent worker action....
A separate module for functions for atomic file operations
Utils is getting too big, so better split this new set of functions intoa separate sub-module. This also allows us to use ResultG there.
Add logging functions lifted into MonadBase
Calling logging functions in a monad transformer stack requiresinstances for all the transformers.This patch adds a module with functions named just as in Ganeti.Logging,but lifted into MonadBase. This allows using them in any transformer...
A quick fix for broken 'make distcheck'
The offending target 'test/hs/offline-test.sh' requires the dependencieseven when the test isn't actually run. This is because currently ourMakefile doesn't distinguish between prerequisites for building testscripts and running them....
Fix tab/spaces indentation in Makefile.am
At some places a few items of lists use 8 spaces, while we use tabseverywhere (as makefiles require anyway). This patch unifies theindentation.
Show Haskell dependencies when compiling
Instead of showing the long GHC command line, show which file is beingcompiled and what its dependencies are. This gives more meaningfulinformation to developers.
Compile every Haskell object file separately
This allows parallel compilation of all targets, including the Haskellobject files. While re-starting GHC for every Haskell source almostdoubles the total CPU time, allowing it run in parallel at the end cuts...
Fix the generation of ListConstants with VPATH
When the build directory differs from the source directory, theautomatic variable '$<' points to the file found by prepending the path.This causes double '../' to be inserted when combined with'$(abs_top_srcdir)'. This patch fixes this....
Add a type class describing types that can serve as locks
Besides the lock order and Show, an additional datum is needed in orderto describe lock inclusion, used, e.g., for group locks. So add a typeclass describing this property.
Disable pylint warnings for ignored files
Some files (like RPC stubs) are marked to be ignored by pylint.However pylint emits warning messages to remind of such files.This disables the warnings.
Signed-off-by: Petr Pudlak <pudlak@google.com>Reviewed-by: Hrvoje Ribicic <riba@google.com>
Add a module for the Ganeti lock structure
Already provide a module holding the structure of the locksavailable in Ganeti, so that the overall structure of WConfDcan be implemented. The actual locks still have to be added.
Verify Basic property for lock allocation
Verify the minimal consistency property for any formof lock handling: if a user holds an exclusive lock,then no other user can hold the same lock (neitherexclusively, nor shared).
Add data structure representing current lock status
To allow for jobs as processes, a central daemon (wconfd) willhandle allocation and release of locks. Add an appropriate datastructure to describe the current status of the locks.
Rename 'BUILT_PYTHON_SOURCES' to 'built_python_sources'
This patch fixes the warnings produced by ./autogen.sh
Makefile.am:310: variable `BUILT_PYTHON_SOURCES' is defined but no program or Makefile.am:310: library has `BUILT_PYTHON' as canonical name (possible typo)...
Add the Python client for WConfD
The client combines the abstract client class and the WConfDstub to provide a Python interface to WConfD.
Generate WConfD Python stubs from Haskell
The generation is included in hs2py.hs, together with otherHaskell to Python code generators.
Generate Python stubs by inspecting functions using TH
The names of the arguments of generated Python methods are derived fromHaskell types to be as descriptive as possible.
Update Makefile.am to properly install generated stubs
This involves adding a new sets of variables for the directory.
Add a Python directory for RPC generated stubs
Directory "lib/rpc/stubs" will contains RPC stubs generated fromHaskell.
Add the WConfD daemon itself
The daemon exposes the declared functions in Ganeti.WConfd.Core to RPCclients (currently just 'echo').
Add the WConfD daemon to build configuration files
Also list it in the Haskell datatype, constants, Python constants andtest configuration.
Add a module for WConfd core functions
There are no exported functions yet, except for `echo`, which justreturns its argument back to a client.
Add a monad for running all WConfD functions
This monad encapsulates working with the daemon and client state, aswell as failures and IO operations.
Add a generic RPC module
The main function in this module takes a list of names of functions anduses TH to create an RPC server, represented as a value of type Handler(from Ganeti.UDSServer). This can be readily used to create a daemonthat dispatches RPC requests to the given functions....
Add tests for Ganeti.THH.Types
In particular, test (de)serialization of "OneTuple".
Add common functions for working with TH's Type
In particular, functions for extracting types of arguments from afunction type, and for uncurrying functions with an arbitrary number ofarguments.
Add helper to handle CLIs that optionally spawn several jobs
This helper function detects whether an opcode returned a list of jobs(i.e., a result of the type ht.TJobIdListOnly) and in this case ituses 'ganeti.cli.JobExecutor' to wait for the jobs and determine the...
Make htools and luxid mandatory in the Makefile
Since Ganeti now uses luxid for core operations, it needs to be alwayspresent.
Add a config. conditional for running Haskell coverage
Similarly to PY_UNIT, add HS_UNIT to run Haskell coverage only if wehave the required modules.
Remove the HTOOLS configuration variable
.. and update the code that uses it.
Add 'kvm-ifup-os' to configure instance communication TAP
The script 'tools/kvm-ifup-os' configures TAP network interfaces forfor instances, routing, DHCP server, etc. Note that this script onlyconfigures TAP network interfaces that are used by the instance...
Add metadata daemon
... including, user, group, daemonizing code with command lineoptions, integration with the Snap HTTP server, and logic.
Signed-off-by: Jose A. Lopes <jabolopes@google.com>Reviewed-by: Michele Tartara <mtartara@google.com>
Let Haddock also generate Hoogle documentation
The Hoogle database is generated to doc/api/hs/ganeti.txt when Haddockis run. The database can be used by developers with a locally installedHoogle command line tool or for a web installation of Hoogle.