Synnefo

.. _Changelog:

Unified Changelog file for Synnefo versions >= 0.13

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Since v0.13 most of the Synnefo components have been merged into a single

repository and have aligned versions.

v0.14next

=========

Released: UNRELEASED

Synnefo-wide

------------

* Integrate Pithos tests in continuous integration.

* Change astakosclient to accept AUTH_URL instead of BASE_URL

  ASTAKOS_BASE_URL settings has been removed from Pithos and Cyclades

  and has been replaced with ASTAKOS_AUTH_URL. Both Pithos and Cyclades

  proxy the Astakos services under ASTAKOS_PROXY_PREFIX path.

  ASTAKOS_PROXY_PREFIX by default has a value of '_astakos'.

  More specifically, Astakos' identity service is proxied under

  '_astakos/identity', Astakos' account service is under '_astakos/account'

  and Astakos' ui service is under '_astakos/ui'.

* Add 'mail_admins' handler to 'django.request' logger in order to send email

  notifactions to users listed in 'ADMINS' setting about unhandled exceptions

  in the code.

Astakos

-------

* Changes in project schema:

  * A Project entry is created when submitting an application for a new

    project, rather than on approval. Its state is dependent on the state

    of its `reference' application (current definition). Lock Project rather

    than Chain (the latter is semantically obsolete).

  * Project states "Active - Pending" and "Suspended - Pending" have been

    removed. In management command `project-list', the existence of a pending

    modification is indicated by a non-blank `Pending AppID'.

  * Improve recording of project, application, and membership actions.

* Implement API calls for projects.

* Store the base URL of a component. Deployer should provide it when adding

  a new component. Service endpoints originating from a component are

  expected to match its base URL; otherwise, a warning is issued.

  Re-registration with `snf-component-register' affects both the base and

  the ui URL.

* Changes in resource and quota handling:

  * New resources are registered with unlimited default base quota,

    represented by 2**63-1.

  * Each newly accepted user copies the default value for all resources

    as their own base quota. A base quota is considered 'custom' if its

    value differs from the default.

  * Changing resource's default quota affects the base quota *only* of

    future users.

  * Resource definition got flags 'api_visible' and 'ui_visible',

    replacing flag 'allow_in_projects'. They control whether a user can

    access these resources. The system internally always accounts for

    all resources, and a user can get off quota even for a resource that

    is not visible.

* Remove API call GET /account/v1.0/authenticate in favor of

  POST /identity/v2.0/tokens.

* Export basic statistics about Astakos service from '/admin/stats/detail' API

  endpoint. Access to this endpoint is only allowed to users that belong to

  the Astakos groups that are defined in the

  'ASTAKOS_ADMIN_STATS_PERMITTED_GROUPS' setting. Statistics are also availble

  from 'snf-manage stats-astakos' management command.

* Management commands:

  * Introduced new commands:

     * component-show

     * quota-list (replacing quota, supports various filters)

     * quota-verify (replacing quota)

  * Changed commands:

     * component-add got options --base-url and --ui-url

     * resource-modify --limit became --default-quota

     * user-modify can operate on multiple users with --all and --exclude

     * user-modify --set-base-quota became --base-quota

  * Removed commands:

     * quota

     * resource-import (subsumed by service-import)

     * resource-export-astakos (subsumed by service-export-astakos)

Cyclades

--------

* Major changes to Cyclades networks:

  * Implement 'cyclades_network' service, containing the /networks, /ports,

    /subnets and /floatingips API endpoints under '/network/v2.0'.

	The old /networks API of 'cyclades_compute' (under /compute/v2.0) is

	removed.

  * Implement `snf-manage subnet-{create, list, modify, inspect}' management

    commands for handling of subnets.

  * Implement `snf-manage port-{create, list, remove, inspect}' management

    commands for handling of ports.

  * Add two new settings, 'CYCLADES_FORCED_SERVER_NETWORKS' and

	'CYCLADES_DEFAULT_SERVER_NETWORKS' to control the networks that newly

	created servers will be connected.

* Implement Floating IP addresses, which are IPv4 addresses that can be

  dynamically added and removed to a running server.

  * Add new 'cyclades.floating_ip' resource.

  * Implement 'snf-manage floating-ip-{create,list,remove,attach,detach}'

	management commands to handle floating IPs.

  * Add 'floating_ip_pool' attribute to networks to mark networks that can

    be used as floating IP pools.

* Implement 'resize' server action.

  * Implement the 'resize' server action, to change the flavor of a server.

  Only 'cpu' and 'memory' resizing is supported.

* Compute quotas for CPU and memory of running vms.

  * Change 'cyclades.cpu' and 'cyclades.ram' resources to represent the CPU

    and RAM for running VMs. Total CPU and RAM usage is represented by new

    'cyclades.total_cpu' and 'cyclades.total_ram' resources.

* Refer to Ganeti NICs by their name instead of their index.

  * Make cyclades give a unique name to each Ganeti NIC. NICs are refered by

    their unique name and not by their index inside the VM that are connected

	to.

* Support firewall profile for all NICs of an instance. Change firewall

  settings to be filled with the unique name of the NIC. The affected settings

  are the GANETI_FIREWALL_{ENABLED, DISABLED, PROTECTED}_TAG settings.

* Add accounting for public IP addresses that is accessible via `snf-manage

  ip-list` management command and via the helpdesk app.

* Implement IPv6 only networks.

* Extend servers info API response with 'SNF:fqdn' attribute, and introduce

  CYCLADES_SERVERS_FQDN to set the template for servers FDQN. Remove

  'UI_VM_HOSTNAME_FORMAT' setting.

* Extend servers info API response with 'SNF:port_forwarding' attribute,

  describing port fowarding rules (DNAT) that are applied to vms. The

  description of such rules is done via the new CYCLADES_PORT_FORWARDING

  setting.

* Speed up server reconciliation, by performing parallel reconciliation for

  each backend.

* Change --dhcp option of network management commands from a flag to a boolean

  value, e.g. --dhcp=True

* Remove 'ARCHIPELAGO_BACKENDS' setting used to distinquish between backends

  that hosted only archipelago backends. Instead allocation is based on which

  disk-templates are enabled in each backend.

* Implement 'snf-manage server-remove' management command.

* Move reconciliation of IP pools from 'reconcile-networks' to

  'reconcile-pools'. The IP pool reconciliation does not reconcile the IP

  pools with Ganeti. Instead it checks if the pool is consistent with the

  IPs that are used by instances.

* Do not automatically release externally reserved IPs if they are released

  from a Ganeti backend. Management of externally reserved IPs must be

  performed from Cyclades with 'network-modify' command.

* Export basic statistics about Cyclades Service from '/admin/stats/detail'

  API endpoint. Access to this endpoint is only allowed to users that belong

  to the Astakos groups that are defined in the 'ADMIN_STATS_PERMITTED_GROUPS'

  setting.  Statistics are also availble from 'snf-manage stats-cyclades'

  management command.

* Support enforcing quota through command 'enforce-resources-cyclades'.

* Remove command 'resource-export-cyclades' subsumed by

  'service-export-cyclades'.

* Obsolete PUBLIC_USE_POOL setting, since Cyclades manages IP pool for all

  type of networks.

* Encrypt / decrypt the instance id / hostname in the  stats URL in

  snf-cyclades-app and snf-stats-app, using the 'CYCLADES_STATS_SECRET_KEY'

  and 'STATS_SECRET_KEY' respectively.

* Add support for snf-vncauthproxy-1.5 and the setting

  'CYCLADES_VNCAUTHPROXY_OPTS', which configures the extra options / arguments

  needed by the newer version of snf-vncauthproxy. Support for older versions

  of snf-vncauthproxy has been dropped. See also the upgrade notes for Synnefo

  and snf-vncauthproxy-1.5.

Pithos

------

* Refactor metadata schema (table attributes) in Pithos DB to speedup current

objects by domain attribute. This is used by Plankton for listing VM images.

* Enforce container-level atomicity in (most) Pithos API calls.

* Remove command 'resource-export-pithos' subsumed by 'service-export-pithos'.

.. _Changelog-0.14.10:

v0.14.10

=======

Released: Tue Nov 26 11:03:37 EET 2013

Cyclades

-------

* This is the first release to support Ganeti 2.8. Support for older versions

  of Ganeti is dropped.

* Use Ganeti opportunistic locking to achive parallelized instance creations

  in the same backend. Add setting 'GANETI_USE_OPPORTUNISTIC_LOCKING' to

  enable the use of this feature.

* Fix warning message while getting object permissions to appear only when

  path is None and the object has permissions

* Add name to newly created NICs and the corresponding firewall tags.

.. _Changelog-0.14.9:

v0.14.9

=======

Released: Mon Nov 11 12:13:31 EET 2013

* Astakos: Fix minor problems with logging in the Astakos module, which could

  lead to unexpected exceptions

.. _Changelog-0.14.8:

v0.14.8

=======

Released: Fri Nov  8 17:25:08 EET 2013

Synnefo-wide

------------

* This is the first release to support Debian Wheezy along with Squeeze. You

  can mix and match nodes freely.

* Update Django dependency to Django>=1.2,<1.5. Django 1.4.5 is available for

  Squeeze through squeeze-backports.

* Since this version, Synnefo ships an example Gunicorn configuration file

  that is automatically installed at ``/etc/gunicorn.d/synnefo.example``.

Cyclades

--------

* On VM creation, pass the hashmap of the image (pithosmap://) instead of the

  image URL (pithos://). Access to the Pithos DB by Ganeti nodes is no longer

  required.

* Workaround race between server creation and server deletion. This will be

  fixed properly by updating Ganeti to support the 'depends' attribute for

  OP_INSTANCE_REMOVE.

Astakos

-------

* For Shibboleth logins, store all attributes along with the user in the DB.

.. _Changelog-0.14.7:

v0.14.7

=======

Released: Wed Sep 18 17:50:12 EEST 2013

Cyclades

--------

* Fix bug in helpdesk view

.. _Changelog-0.14.6:

v0.14.6

=======

Released: Wed Sep 18 16:18:58 EEST 2013

Pithos

------

* Substitute the PITHOS_BACKEND_QUOTA setting with two

  distinct settings: PITHOS_BACKEND_ACCOUNT_QUOTA &

  PITHOS_BACKEND_CONTAINER_QUOTA

* Set PITHOS_BACKEND_CONTAINER_QUOTA default value to 0 (unlimited)

* Fix bug that resulted in DB deadlocks.

Cyclades

--------

* Fix bug in snf-dispatcher that resulted in servers to be deleted from the

  DB even if the corresponding Ganeti job failed.

Branding

--------

* Add new BRANDING_FOOTER_EXTRA_MESSAGE setting.

.. _Changelog-0.14.5:

v0.14.5

=======

Released: Wed Aug  7 11:19:49 EEST 2013

Pithos

------

* Fix security issue with handling Pithos versions.

.. _Changelog-0.14.4:

v0.14.4

=======

Released: Mon Jul 29 12:24:22 EEST 2013

Pithos

------

* Fix bug in reconcile resources management command.

.. _Changelog-0.14.3:

v0.14.3

=======

Released: Thu Jul 25 12:22:47 EEST 2013

Synnefo-wide

------------

* Use the SYNNEFO_TRACE environmental variable to control whether the greenlet

  tracing code will get loaded or not.

* Split the HIDDEN_COOKIES setting in HIDDEN_HEADERS and HIDDEN_COOKIES, and

  add the MAIL_MAX_LEN setting, to limit the mail size for unhandled

  exceptions.

.. _Changelog-0.14.2:

Released: Fri Jul 12 13:13:32 EEST 2013

v0.14.2

=======

Cyclades

--------

* Add new setting PITHOS_BACKEND_POOL_SIZE, which configures the size

  of the pool of Pithos backends that are used by plankton.

.. _Changelog-0.14:

v0.14

=====

Released: Tue Jun 25 14:01:19 EEST 2013

Synnefo-wide

------------

* Create 'snf_django' Python package to hold common code for all Synnefo

  components.

* Create a JSON-exportable definition document for each Synnefo Components

  (Astakos, Cyclades, Pithos, etc.) that consolidates APIs (services),

  resources, and other standardized properties (e.g. default URL prefixes).

* Standardize URLs for Synnefo Components, impose structure and naming

  conventions to related settings. Make each component deployable under

  a user-configurable <COMPONENT>_BASE_URL. Each API (compute, image, etc.)

  is deployable under a developer-configurable prefix beneath BASE_URL.

* Deprecate CLOUDBAR_ACTIVE_SERVICE setting from all apps. 

* Common synnefo 404/500 templates (located in snf-webproject)

Astakos

-------

* Redesign of the accounting system (quotaholder) and integration into

  Astakos.

  * Simplified the quotaholder model; removed tables Entity and Policy; now

    table Holding contains limit and usage for every holding.

  * Extended table Holding, so that we can keep track of quota for every

    valid combination of holder (e.g. user), resource, and source (e.g. the

    default system or some specific project).

  * Refactored code for issuing and resolving commissions for robustness;

    added a 'force' option to bypass the upper limit check when issuing a

    commission.

  * Simplified syncing to the quotaholder; removed fields from models

    Project and ProjectMembership, previously needed for syncing; removed

    state PROJECT_DEACTIVATED from ProjectMembership.

  * Removed settings ASTAKOS_QUOTAHOLDER_URL, ASTAKOS_QUOTAHOLDER_TOKEN,

    and ASTAKOS_QUOTAHOLDER_POOLSIZE.

* API-related changes:

  * Implemented API calls for quota, resources, and commissions.

  * Moved all API calls under '/account/v1.0'.

  * Implemented the keystone API call POST /tokens under '/identity/v2.0'.

* Service and resource specification and handling:

  * Specified a format for defining services along with the API endpoints

    and the resources they expose. Migrated internal resource name by

    prefixing it with service name (e.g. 'vm' becomes 'cyclades.vm');

    renamed registered service 'pithos+' to 'pithos'.

  * Specified a procedure to register a Synnefo component, its services and

    their resources in astakos and set the resources' default base quota

    limit. Removed resource definitions from settings.

  * Moved service and resource presentation data out of the respective db

    models into a separate file of UI constants.

* Converted the limit on pending applications from a setting to a quotable

  resource. Converted the related user setting to a user-specific base quota

  limit. Deprecated model UserSetting; removed setting

  ASTAKOS_PENDING_APPLICATION_LIMIT.

* Changes in locking strategy:

  * Lock only project's chain for all project operations; lock user before

    syncing to quotaholder.

  * When locking multiple rows (e.g. users or holdings) include an ORDER BY

    clause in the query to impose ordering on locking.

* Changes in views:

  * Replaced custom transaction context with a simple decorator for managing

    transactions and a context 'ExceptionHandler', which logs and suppresses

    exceptions

* Added fine grain user auth provider's policies.

  * Administrator can override default auth provider policies to a specific 

    user or group of users.

  * Optionally a user can be assigned to a list of groups, based on the

    authentication method he choosed to signup.

* Removed explicit handling of SMTP errors on each email delivery. Exceptions 

  are now propagated to base django exception handler.

* Email used in html/email tempaltes which prompt user to contact for service 

  support prompts is now defined in ``CONTACT_EMAIL`` setting introduced in 

  snf-common settings.

* Improvements in user activation flow

  * User moderation now takes place after the user has verified his email

    address.

  * User model enriched with additional user state fields

  * Split activation email from moderation process. Administrator is required 

    to moderate user explicitly using the `user-modify --accept` or

    `user-modify --reject` commands.

  * Improved logging throught out user activation procedures.

* Remove deprecated AstakosUser model fields: `provider`,

  `third_party_identifier`

* Allow override of authentication provider messages using the following 

  format in setting names: ``ASTAKOS_<PROVIDER_MODULE>_<MSGID>_MSG``

* Cloudbar automatically tries to identify the active service based on window 

  location.

* Removing authentication provider view is now CSRF protected.

* New `API access` view, containing useful information to users on how to 

  access available Synnefo services API's.

* Remove of ASTAKOS_*_EMAIL_SUBJECT settings. All email subjects are now 

  defined in astakos.im.messages module. Overriding default values can be 

  achieved using custom gettext files or using astakos messages settings::

	#change of greeting email subject

  	ASTAKOS_GREETING_EMAIL_SUBJECT_MESSAGE = 'Welcome to my cloud'

* Remove ``ASTAKOS_ACTIVATION_REDIRECT_URL`` and ``ASTAKOS_LOGIN_SUCCESS_URL`` 

  from astakos .conf file. Settings are dynamically computed based on 

  ``ASTAKOS_BASE_URL``.

* Management commands:

  * Introduced new commands:

     * authpolicy-{add, list, remove, set, show}

     * group-{add, list}

     * component-{add, list, modify, remove}

     * reconcile-resources-astakos

     * resource-{export-astakos, import, modify}

     * service-{export-astakos, import, show}

  * Renamed commands:

     * astakos-quota to quota

     * user-update to user-modify

     * full-cleanup to cleanup-full

  * Removed commands:

     * astakos-init

     * invitation-{details, list}

     * project-sync

     * resource-{add, remove}

     * service-{add, remove, token-renew, update}

     * user-invite

     * user-set-initial-quota (integrated its functionality in user-modify and quota)

  * Added quota and project-related information in user-show command; added

    membership information in project-show.

Cyclades

--------

* Make 'type' attribute required for network create API request.

* Networks not created to all Ganeti backends upon creation, they are instead

  created to a backend only when a VM connects to the network.

* Add 'CYCLADES_ASTAKOSCLIENT_POOLSIZE' setting which tunes the size of the

  http connection pool to astakos.

* Remove 'CYCLADES_USER_CATALOG_URL' and 'CYCLADES_USER_FEEDBACK_URL' settings

* Remove CYCLADES_USE_QUOTAHOLDER, CYCLADES_QUOTAHOLDER_TOKEN,

  CYCLADES_QUOTAHOLDER_URL, CYCLADES_QUOTAHOLDER_POOLSIZE settings

* Rename 'cyclades-usage-verify' management command to

  'reconcile-resources-cyclades'. Also, remove 'cyclades-usage-reset' command,

  which is equivalent to 'reconcile-resources-cyclades --fix'.

* Rename 'cyclades-reconcile-commissions' management command to

  'reconcile-commissions-cyclades'.

* Remove obsolete 'MAX_VMS_PER_USER', 'MAX_NETWORKS_PER_USER',

  "VMS_USER_QUOTA" and "NETWORKS_USER_QUOTA" settings, since their usage

  is covered by Quotaholder.

* Remove obsolete setting 'API_ROOT_URL', since it is being covered by

  the use of CYCLADES_BASE_URL* Remove obsolete setting 'API_ROOT_URL', since

  it is being covered by 'CYCLADES_BASE_URL'.

* Remove obsolete settings GANETI_DISK_TEMPLATES and

  DEFAULT_GANETI_DISK_TEMPLATE

Cyclades helpdesk

-----------------

* Additional start/stop vm action

* Display extend backend info in vm's view

* Fixed IP lookup

Pithos

------

* Remove PITHOS_AUTHENTICATION_USERS setting, which was used to override

  astakos users.

* Remove 'PITHOS_USER_CATALOG_URL', 'PITHOS_USER_FEEDBACK_URL' and

  'PITHOS_USER_LOGIN_URL' settings.

* Remove PITHOS_USE_QUOTAHOLDER, PITHOS_QUOTAHOLDER_URL,

  PITHOS_QUOTAHOLDER_TOKEN and PITHOS_ASTAKOSCLIENT_POOLSIZE

Tools

-----

.. _Changelog-0.13:

v0.13

=====

Released: Wed Apr 10 18:52:50 EEST 2013

In v0.13 the code was very heavily refactored for increased uniformity since

most of the Synnefo components have been merged into a single repository. Thus,

**just for this version** we will not document a complete Changelog (features,

fixes, improvements, issues, setting changes), but rather just copy from the

`NEWS` file with minor additions wherever needed.

Synnefo-wide

------------

* Support for pooling throughout Synnefo

  * Pooled Django DB connections, Pithos backend connections, HTTP

    connections using single `objpool` package

* Improved management commands

  * Unified codebase for output of tables in JSON, CSV

* Bring most of Synnefo code inside a single, unified repository

  * support automatic Python and Debian package builds for individual commits

  * with automatic version generation

* Overhauling of Synnefo settings: renames and refactoring, for increased

  uniformity (in progress)

* Deployment: Standardize on gunicorn, with gevent-based workers

  and use of Green threads throughout Synnefo

* Documentation: New scale-out guide, with distinct node roles,

  for mass Synnefo deployments

Astakos

-------

* Support multiple authentication methods

  * Classic (username/password), Shibboleth, LDAP/Active Directory,

    Google, Twitter, LinkedIn

  * Users can enable/disable auth methods, and switch between them

* Introduce a UUID as a global identifier for users, throughout Synnefo

  * The UUID remains constant as the user enables/disables login methods

* Allow users to modify their email address freely

* Per-user, per-resource accounting mechanism (quotaholder)

* Full quota support, with per-user, per-resource quotas, based on quotaholder

* Projects: Users can create and join Projects

  * Projects grant extra resources to their members

* UI Enhancements for quotas and projects

  * distinct Usage tab, showing usage of individual resources

  * Project management UI

  * New Overview page

* refactored/improved /login endpoint used by desktop/mobile clients.

  * endpoint url is now exposed by `weblogin` service

  * clients should use unauthenticated identity/tokens api to resolve the 

    endpoint url

  * view only allows redirects to `pithos://` scheme urls

  * removed uuid from redirect parameters. Client should use authenticated 

    request to identity/tokens to retrieve user uuid.

Cyclades

--------

* Commission resources on quotaholder/Astakos

* Support mass creation of flavors

* Support for the ExtStorage disk template in Ganeti

* Query and report quotas in the UI

* Pass VM configuration parameters over a VM-side API (`vmapi`)

  * Do not pass sensitive data as Ganeti OS parameters

  * Keep sensitive data in memory caches (memcached) and

    never allow them to hit the disk

* Display additional backend information in helpdesk machines list

* Allow helpdesk users to search for an account using a known machine id

* Helpdesk actions are now logged using the synnefo's common login

  infrastructure

UI

^^

* Removed feedback endpoint. Feedback requests delegate to astakos feedback

  service. ``FEEDBACK_CONTACTS``, ``FEEDBACK_EMAIL_FROM`` settings removed, 

  and no longer used.

* ``UI_LOGIN_URL``, ``UI_GLANCE_URL``, ``COMPUTE_URL`` settings no longer 

  required to be set and are dynamically computed based on ``ASTAKOS_BASE_URL``

  and ``CYCLADES_BASE_URL`` settings.

* File group is no longer included in ssh keys personality metadata sent in 

  create vm calls.

Pithos

------

* Support storage of blocks on a RADOS backend, for Archipelago

  * new settings:

    PITHOS_RADOS_STORAGE, PITHOS_RADOS_POOL_BLOCKS, PITHOS_RADOS_POOL_MAPS

* X-Object-Public now contains full url (domain + proper component prefix + 

  file path)

* Rewritten support for public URLs, with admin-selectable length

  * new settings:

    PITHOS_PUBLIC_URL_SECURITY, PITHOS_PUBLIC_URL_ALPHABET

* Enable pithos backend to use external quotaholder component

  * new settings:

    PITHOS_USE_QUOTAHOLDER, PITHOS_QUOTAHOLDER_URL, PITHOS_QUOTAHOLDER_TOKEN,

    PITHOS_QUOTAHOLDER_POOLSIZE

* Moderated version debiting mechanism

  * new setting:

    PITHOS_BACKEND_FREE_VERSIONING

* Proxy Astakos user-visible services

  * new settings:

    PITHOS_PROXY_USER_SERVICES, PITHOS_USER_CATALOG_URL,

    PITHOS_USER_FEEDBACK_URL, PITHOS_USER_LOGIN_URL

Tools

-----

* Extend snf-burnin to include testing of Pithos functionality