OS images in 'LUInstanceCreate' and OS scripts optional
Instance create with OS image
Extend 'LUInstanceCreate' to image the instance's first disk if an OSimage is specified via the OS params. If the OS image is a file, itwill be copied to the node via SSH. If ths OS image is a URL, it willbe passed directly to the node, which will then download the file....
Function to check if the OS image parameter is valid
Add helper function to check if the OS image parameter contained inthe OS parameters of an opcode is valid.
Signed-off-by: Jose A. Lopes <jabolopes@google.com>Reviewed-by: Hrvoje Ribicic <riba@google.com>
Function to image disks while ensuring that disks are paused
Function to remove instance if disks are degraded
Signed-off-by: Jose A. Lopes <jabolopes@google.com>...
RPC 'blockdev_image' to image devices
Add RPC 'blockdev_image' that uses 'ganeti.backend.BlockdevImage' todump an image to an instance's disk device, optionally downloadingthat image.
Helper function to image a device by downloading or dumping
Add 'BlockdevImage' which downloads a file and dumps it to aninstance's disk if the path is a URL, otherwise it dumps the filedirectly to the instance's disk.
Helper function that downloads an image and dumps it to disk
Generalize 'WipeDevice' to 'DumpDevice'
Helper functions to get and update OS image from OSParams
Add 'GetOSImage' and 'PutOSImage' which handle the OS image key in theOS parameters dict.
Fix export order according to definition order
Fix docstrings
Fix several docstrings.
Reuse method to parse name from OS 'name+variant' string
Support releasing locks by level
Add a function to WConfD allowing to release locks byGaneti lock level.
Signed-off-by: Klaus Aehlig <aehlig@google.com>Reviewed-by: Petr Pudlak <pudlak@google.com>
Add a convenience function to release locks by property
While definable from listLocks and updateLocks, it is convenientto be able to all locks with a given property. This will, e.g.,allow to release all locks at a given level---as the mcpu doesroutinely....
BGL is not a set lock
Despite what the name suggests, the Big Ganeti Lock does not act as aset of all locks. It is just a lock as any other lock from the pointof view of the Ganeti lock hierarchy.
It's special meaning comes from the fact that all jobs, except for...
Verify serialisation of LockLevel
For the new JSON instance Ganeti.Locking.Locks.LockLevel, verify thatreadJSON . showJSON = Ok.
For each Ganeti Lock provide its level
...so set a job can request operations on all its locks of a given level.
Add network locks to Haskell
Ganeti also has a lock level for networks, however this level was forgottenwhen the Ganeti lock hierarchy was implemented in Haskell.
Fix allocation order of locks
The correct allocation order for node-related Ganeti locks isallocation, group, the node itself, resources.
Allow transitions from sets to elements
Owning a set-lock has the same effect as owning each member. So askingfor a member while already holding the set lock in the same mode (sharedor exclusive) is, effectively, no asking for new locks. Therefore, do not...
Fix typo
Merge branch 'stable-2.11' into master
Merge branch 'stable-2.10' into stable-2.11
Merge branch 'stable-2.9' into stable-2.10
Fix error introduced during merge
A parameter was lost while resolving a conflict in the signature of a function.
Signed-off-by: Michele Tartara <mtartara@google.com>Reviewed-by: Klaus Aehlig <aehlig@google.com>
gnt-cluster copyfile: accept relative paths
If, on the command line, the argument to gnt-cluster copyfile isa relative path, consider this a shorthand for the correspondingabsolute path.
Signed-off-by: Klaus Aehlig <aehlig@google.com>Reviewed-by: Michele Tartara <mtartara@google.com>
Merge branch 'origin/stable-2.8' into stable-2.9
Let WConfd distribute SSConf to nodes
.. and remove the corresponding code from lib/config.py.
Signed-off-by: Petr Pudlak <pudlak@google.com>Reviewed-by: Klaus Aehlig <aehlig@google.com>
Add an asynchronous task for distributing SSConf to nodes
The task computes the SSconf data and compares it to the last state thatwas successfully distributed. If it differs, issues the RPC call todistribute it. If any of the RPC calls fails, the state remains...
Add Haskell RPC call for distributing SSConf to nodes
The data is sent as a dictionary mapping file names to lists of lines.
Let the SSConf RPC server side handle lists
Since on Haskell side we represent SSConf as list of lines, let thePython side understand it as well.
Add a test for (de)serialization of Ssconf
.. to verify that readJSON . showJSON = Ok.
Add a module for constructing Ssconf from the configuration
This migrates the functionality from PythonConfigWriter._UnlockedGetSsconfValues to Haskell.
Add a data type for representing the complete SSConf
.. as a map from SSKeys to lists of strings.
Improve Haskell functions for managing the UidPool
Add a formatting function that is needed by Ssconf.Also expose 'ipFamilyToRaw', which is needed as well.
Add missing keys to Ganeti.Ssconf
These keys were probably introduced only in Python later and weren'tadded to the Haskell code.
Let WConfd distribute the configuration to MCs
.. and remove the distribution from lib/config.py
Function that distributes the configuration to MCs
Also refactor helper functions in WConfd/ConfigWriter into smaller onesand add helpers for RPC calls.
Add a function for listing online nodes to Haskell
.. and also simplify 'getMasterCandidates'.
Add the RPC call for uploading a file to Haskell
.. and also add a function that prepares the call from a given file.
Functions for conversions between POSIX time and ClockTime
This will be needed by the RPC calls to convert Haskell nativerepresentation to the one used in RPC.
Add reverse mappings to RuntimeEnts from UIDs to names
.. and GIDs to names. Also create a proper data type for it.
The reverse mapping to names is needed for properly translating localIDs to names for RPC calls.
Use 'ResultT' for 'getEnts' and handle all exceptions, not just when a...
Add a new RPC server call for uploading a single file
The server side processes the request exactly the same as for"upload_file".
Unlike "upload_file", the new call "upload_file_single" declares allrequired fields without requiring additional preprocessing....
Add more meaningful error messages to asserts in vcluster
.. to simplify debugging of RPC calls.
Create a data type for representing RPC compressed data
It is serialized as a JSON pair. The first element is a numberdescribing the compression type, the second element the actual data(in this case Base64+Zlib encoded).
On the Python side it is decompressed by backend._Decompress....
Add a module for handling compression
Since the Haskell 'zlib' library by default uses exceptions forsignalling errors, we need to use its internals to implement a proper,pure decompressing function.
Enhance logRpcResults to also return the list of errors
It is common to log all RPC errors and then check if there were any.Since 'logRpcResults' needs to filter the errors anyway, let it returnit.
Factor out the part that filters out errors into a separate function....
Use lenses for modifying DaemonState in WConfd
This removes the verbose code for record updating and leads to morepoint-free code.
Write the configuration only if it changed
.. to avoid expensive operation when they're not actually needed.Comparing the two objects in memory is much less costly thatdistributing the configuration.
Add a module for generated lenses for Ganeti.Objects
The module also adds type classes for lenses for common fields (serial,mTime, uuid and tags).
Splitting the lenses into a separate module avoids problems with unusedfunctions.
Signed-off-by: Petr Pudlak <pudlak@google.com>...
Add a module for custom generic Lens functions
The module currently re-exports Control.Lens and adds a function thatcreates lenses from data types by appending 'L' to its field names.
All Ganeti modules should import this module instead of Control.Lens...
Add 'lens' Haskell dependency
.. that will be used in WConfd
Add a helper for creating 'FileMode' fields
.. which will be used for RPC objects.
Move helpers for specific fields into a separate module
These helpers aren't needed by other code in Ganeti.THH and since themodule is already large, it'll be better to separate them into theirown.
No functionality is changed by this patch, just functions moved....
Add a helper to ConfigWriter for creating async. workers
In particular, stateful workers that can fail with an error, in whichcase the error is logged and the state remains unchanged.
Add Foldable/Traversable instances for GenericContainer
This makes working with it easier as it allows use of many standardfunctions.
Improve RAPI detection of the watcher
If the watcher is not allowed to access RAPI, it doesn't mean that it is deadand needs to be restarted.
Fixes Issue 752.
Signed-off-by: Michele Tartara <mtartara@google.com>Reviewed-by: Hrvoje Ribicic <riba@google.com>
Update style guide with respect to Haskell imports
With the number of imports increasing, our style hasevolved over time.
- We don't distinguish between standard and non-standard non-Ganeti libraries any more.
- To help the reader identify which library a function is...
Clean up RunWithLocks
This patch cleans RunWithLocks up a little bit by reducing the numberof delay function terminations, and using the QAThread class to ensureexceptions are thrown at the right time and in the right place.
Signed-off-by: Hrvoje Ribicic <riba@google.com>...
Add an exception-trapping thread class
To have better control over threads, this patch adds a helper Threadsubclass which captures any exceptions occurring for later use.
Signed-off-by: Hrvoje Ribicic <riba@google.com>Reviewed-by: Petr Pudlak <pudlak@google.com>
Wait for delay to provide interruption information
The RunWithLocks test assumed that gnt-debug delay would have the infoneeded for interruption ready immediately after being run, and in somesituations this is not the case. This patch makes the test more patient...
Add patching QA configuration files on buildbots
Enable a timeout for instance shutdown
Add the timeout parameter to the StopInstance function of the hypervisor baseclass and to all its implementations.
Also, change the tests as required by this change.
Signed-off-by: Michele Tartara <mtartara@google.com>...
Allow KVM commands to have a timeout
Modify the function that sends commands to the KVM monitor so that it ispossible to specify an optional timeout after which the command is killed.
Allow xen commands to have a timeout
Modify the function that runs Xen commands so that it is possible to specify anoptional timeout after which the command is killed.
Fix wrong docstring
Fields must be the final elements in an epytext string.
Also clean up stale livelock files
Upon detection of a stale livelock, make WConfD alsotry to clean it up. We can safely ignore any failures,as the Ganeti resources are already freed, and the worstthing that can happen is an empty file left over in the...
Add Haskell dependency on 'lens'
.. which will be needed soon for WConfd.
While at it, also ensure we have the required version ofbase64-bytestring.
Refactor warnings issued by the latest 'hlint'
.. as the newest version of hlint adds additional checks
devel: Instal vanilla GHC7 in the squeeze schroot
Since the default GHC6 isn't usable for modern libraries and startingfrom wheezy there is GHC7 anyway, download and install the currentvanilla version as well as the latest Cabal.
Use the current up-to-date libraries....
devel: Make the built schroots rsync-able, if available
This speeds down transferring them to buildbots, at a negligible sizecost.
Use correct lockfile for gnt-debug wconfd
As jobs are currently running in masterd, use the masterd livelockfile.
Add utility to guess livelock file for an owner
As livelock files are constructed in a systematic manner,we can guess what the livelock file for a given owner is.While this will not necessarily work perfectly, it will beuseful to simplify direct debugging of WConfD....
Make masterd create a livelock file
...so that it can request resources from WConfd.
Rename setup_queue to setup_context in masterd
...as this function sets up a much richer context than justthe job queue, including the current lock management.
Add utilities for liveliness lock files
To request resources from WConfD, requesters have to providethe name of a file they own an exclusive lock on. In this way,their death can be detected. Add utility functions to obtainsuch a file name.
Signed-off-by: Klaus Aehlig <aehlig@google.com>...
Ensure the existence of LIVELOCK_DIR
Add a path to store the lock files presented to WConfD
When requesting resources from WConfD, a file has to bepresented where an exclusive lock is owned on, so thatWConfD can detect when the requester dies. Add a pathto a directory where these files are kept in....
In WConfD run a cleanup task
Add the cleanup task for locks of died owners to WConfD,as a separate thread.
Provide a periodic lock cleanup task
Provide a task that periodically checks for all locksowners whether they are still alive. If not, it cleansup their locks.
Add a constant for the polling of jobs death
Add a constant for the time interval at which WConfD checks forresource owners that have died and cleans up their resources.
Verify that all lock owners indeed own a lock
Add a test verifying that the list of lock owners is nottoo big in the sense that every subject contained in thatlist owns at least one lock.
Verify the list of lock owners is complete
In other words, verify that if all the lock owners release theirlocks, the empty lock allocation is reached.
Add a function to determine the lock owners
To allow for clean up, add a function that provides the listof all the owners holding at least one lock.
Add a function to detect death of a job
With jobs running in processes different from the lock management soon,we need to detect if a job holding locks dies, in order to release theresources.
Convert int to float when checking config. consistency
When reading the configuration file from RPC JSON, values without afloating point are parsed as 'int', not as 'float', and later theconsistency check fails.
This patch adds an automatic conversion from 'int' to 'float' during...
Update the NEWS file
Add an expected block option to RunWithLocks
To compensate for the cases where a QA test is supposed to block whena lock is present, add an additional option showing whether blocking issupposed to happen or not.
Track if a QA test was blocked by locks
This patch adds threading to the RunWithTests function, allowing onethread to execute the QA test, and the other to monitor if it is beingblocked by locks set up during the test. If it is, terminate theblocking job, and let the QA continue, reporting the test failure at...
Add option to turn off command logging to GetCommandOutput
The command logging performed by GetCommandOutput can be distractingand unwelcome. This patch adds a switch allowing the command output tobe muted as needed.
Add an option for disabling the SSH multiplexer in QA
The SSH multiplexer is used by default in every connection made, andwhen multiple threads are used, this causes strange issues when runningthe QA locally. This patch allows the multiplexer to be bypassed,...
Add a RunWithLocks QA utility function
This patch adds a QA utility function that acquires a set of locks, andattempts to run a given function with the locks in place. Should thegiven function block, this function does not detect this - laterpatches will address the issue....
Sort imports in qa_rapi.py
This patch sorts the imports of qa_rapi.py to make things a little bitmore orderly.
Align timestamps in gnt-job info
This patch aligns the timestamps output as a part of gnt-job info, andperforms minor refactorings in the process.
Add alignment support to PrintGenericInfo
Aligning dictionary entries makes no difference to a YAML parser, butmakes the output much easier to read and compare. This patch adds thepossibility of specifying alignment groups to ordered dictionaryentries....
Make gnt-job info output valid YAML
This patch changes gnt-job info to use standard functions defined incli.py, and output valid YAML.
Make PrintGenericInfo handle tuples better
The PrintGenericInfo function in cli.py did not handle tuples ascontainers of items, making it impossible for these to be deserializedautomatically when a YAML parser is used. This patch adds separatehandling of tuples, including inlining them for readability when...
Refactor move-instance QA test
The move-instance QA test will have to be changed in the followingpatches to allow testing opportunistic locking.
This patch retains the same functionality as before, but allows aniallocator to be used, and splits the move back and forth into two...