\usepackage{color}
\newcommand{\cL}{{\cal L}}
+\newcommand{\TODO}{{\sl TODO \marginpar{\sl TODO}}}
\begin{document}
\conferenceinfo{ScalaDays '12}{London, UK.}
\title{Aquarium: Billing for the Cloud in the Cloud}
-\authorinfo{Georgios Gousios \and Christos Loverdos}
+\authorinfo{Georgios Gousios \and Christos KK Loverdos \and Nectarios Koziris}
{GRNet SA}
-{\{gousiosg,loverdos\}@grnet.gr}
+{\{gousiosg,loverdos,nkoziris\}@grnet.gr}
\maketitle
\begin{abstract}
This paper describes the architecture for the Aquarium cloud infrastructure
- software.
+ software. Aquarium is a new software we have built whose main function is to associate cloud resources usage with charging policies.
\end{abstract}
\category{D.3.3}{Programming Languages}{Language Constructs and Features}[Control structures]
\section{Introduction}
\section{Requirements}
+Aquarium was designed on a clean sheet to serve a particular purpose,
+namely the provision of billing services to an IaaS infrastructure,
+and also be extensible to new services. In the following sections,
+we briefly present the requirements that shaped Aquarium's design.
\subsection{Application Environment}
-Aquarium is designed and developed as part of the Okeanos project at GRNet. The
+Aquarium developed as part of the Okeanos project at GRNet. The
Okeanos project is building a full stack public IaaS system for Greek
universities, and several services on top of it. Several components comprise
the Okeanos infrastructure:
\end{description}
While all the above systems (and several prospective ones) have different
-user interfaces and provide different functionalities in the context of
+user interfaces and provide distinct functionality in the context of
the GRnet IaaS, they all share a common notion of \emph{resources}, access
and manipulation options to which they offer to users.
+\subsection{Sharing}
+
+The Okeanos IaaS will support the Greek higher education, an estimated
+population of 100.000 students and researchers. Each member will be granted
+access to a collection of resources using her institutional account. To enforce
+a limit to the resources that can be acquired from the platform, each user will
+have a limited amount of credits, renewable each month, which will be allowed
+to spend on any resource available through the infrastructure. Resources can
+also be shared; for example files on the Pithos file service or virtual machine
+images on the Archipelago storage are potentially subject to concurrent usage from
+multiple users. This means that charges for the use of a single resource
+may need to be distributed among several users. Also this may mean that in order
+for sharing to work correctly, users may need to transfer credits among them.
+
\subsection{Configuration}
systems.
-
\subsection{Scaling}
In the context of the Okeanos system, Aquarium provides billing services on a
failure, Aquarium must not corrupt any billing data under any circumstances,
while it should reach an operating state very fast after a service restart.
-\section{Architecture}
+\section{Domain Modeling}
+\subsection{Basic terminology}
+We have already mentioned several entities in our description so far. Let us be a bit more specific on several key terms.
+\begin{description}
+\item[Credits]
+The analog of money. Credits are the `universal money` within Aquarium.
-\section{Implementation}
+\item[Resource]
+A billable/chargeable entity. We generally need credits to use a resource. When a resource is used, then consume credits. Examples of resources are the `download bandwidth` and, respectively, the 'upload bandwidth', the `disk space` and the `VM time` to name a few. Generally speaking, the ``resource'' term specifies the type. A resource may have several properties attached, i.e. it name, unit of measure, a description of how its consumption translates to credit and whether it can have more than one instances.
-\subsection{The configuration DSL}
+\item[Resource instance]
+A user may have several instances of a resource type. For example, regarding a ``Virtual Machine'' resource, a user may have more then one of them. They are distinguished by their unique resource instance identifier. We call resources that can have more than one instance ``complex'' resources.
-The configuration requirements presented above were addressed by creating a new
-domain specific language ({\sc dsl}), based on the YAML format. The DSL
-enables administrators to specify billable resources, billing policies and
-price lists and combine them arbitrarily into agreements applicable to specific
-users, user groups or the whole system.
-The DSL supports inheritance for policies, price lists and agreements and composition in the case of agreements.
-It also facilitates the
-definition of generic, repeatable debiting rules, which are then used by the
-system to refill the user's account with credits on a periodic based.
-
-The DSL is in itself based on five top-level entities, namely:
+\item[Resource event]
+An event that is generated by a system, which is responsible for the resource.
+The resource event describes a state change for the resource. In particular, a resource event records the time when that state change occurred (`occurredMillis` attribute) and the changed value (`value` attribute).
-\begin{description}
+\item[Cost policy]
+A cost policy refers to a resource and it is the policy used in order to charge credits for resource usage. Cost policies come in three flavors, namely \textsf{continuous}, \textsf{discrete} and \textsf{onoff}.
+
+\item[Pricelists] assign a price tag to each resource, within a timeframe.
- \item[Resources] specify the properties of resources that Aquarium knows
- about. Apart from the expected ones (name, unit etc),
- a resource has two properties that affect billing: \textsf{costpolicy}
- defines whether the billing operation is to be performed at the moment
- a billing event has arrived, while the \textsf{complex} attribute defines
- whether a resource can have many instances per user.
-
- \item[Pricelists] assign a price tag to each resource, within a timeframe.
-
- \item[Algorithms] specify the way the billing operation is done in response
- to a billing event. The simplest (and default) way is to multiply the
- billable quantity with the applicable price. To enable more complex billing
- scenarios, the Aquarium DSL supports a simple imperative language with
+\item[Charging algorithms] specify the way a resource event can generate consumed credits.
+A charging algorithm can be as simple as a direct multiplication of the
+ chargeable resource quantity with the applicable price. We offer the option, though, to define more complex charging scenarios, the Aquarium DSL supports a simple imperative language with
a number of implicit variables (e.g. \texttt{price, volume, date})
that enable administrators to specify, e.g. billing algorithms that
- scale with billable volume. Similarily to pricelists, algorithms
+ scale with billable volume. Similarily to price lists, charging algorithms
have an applicability timeframe attached to them.
-
- \item[Crediplans] define a number of credits to give to users and a repetition
+
+\item[Credit plans] define a number of credits to give to users and a repetition
period.
- \item[Agreements] assign a name to algorithm, pricelist and creditplan triplets,
+\item[Agreements] assign a name to a charging algorithm, pricelist and creditplan triplets,
which is then assigned to each user.
+
+
+\item[Resource instance state]
+A value which is associated with a resource instance. Usually a floating point number, as in the $10.5$ MB designation regarding a possible ``total current downloading bandwidth''.
+
+\item[User]
+An owner of resources and credits. Users are defined externally of Aquarium.
+
+\item[Resource event store]
+A datatabase, in the general sense, where resource events are stored.
+
+\item[User bill]
+A, usually, periodic statement of how many credits the user has consumed. It may contain detailed analysis that relates consumed credits to resources.
+
+\item[Billing period]
+A time period at the end of which we issue a user bill.
+A billing period is made of a starting date and a duration that is a multiple of a week.
+A usual billing period starts on a particular month date (eg. 3rd) and lasts for a month.
+Each resource type designates what happens to its accumulated value (if any) at the beginning of the billing period. Usually, at the beginning of the billing period, the accumulating amounts of resources are set accumulating amount. For example, for a monthly billing period, the total uploading bandwidth is reset to zero every month.
+
+\item[User state]
+The user state is made of the following distinct parts, of which the first two can be integrated to a unifying ``resource'' concept.
+
+\begin{enumerate}
+\item User credit state, that is the total credit amount for the user.
+
+\item User resource state, which refers to the state of each resource instance that the user owns.
+
+\item Processing state \TODO
+\end{enumerate}
+
+\item[Resource event processing]
+The set of algorithmic steps by which a resource event leads to state changes of the user state.
\end{description}
+\subsection{The configuration DSL}
+\label{sec:dsl}
+The configuration requirements of Aquarium were addressed by creating a new
+domain specific language ({\sc dsl}), based on the YAML format. The DSL
+enables administrators to specify chargeable resources, charging policies and
+price lists and combine them arbitrarily into agreements applicable to specific
+users, user groups or the whole system.
+The DSL supports inheritance for policies, price lists and agreements and composition in the case of agreements.
+It also facilitates the
+definition of generic, repeatable debiting rules, which are then used by the
+system to refill the user's account with credits on a periodic base.
+
+From the previously specified term, the following five are used in the DSL:
+
+\begin{enumerate}
+\item Resources
+\item Charging algorithms
+\item Pricelists
+\item Credit plans
+\item Agreements
+\end{enumerate}
+
+
\begin{figure}
-\lstset{language=ruby, basicstyle=\footnotesize,
+\lstset{language=c, basicstyle=\footnotesize,
stringstyle=\ttfamily,
flexiblecolumns=true, aboveskip=-0.9em, belowskip=0em, lineskip=0em}
repeat:
- start: "00 02 * * Tue"
end: "00 02 * * Wed"
- from: 1326041177 #Sun, 8 Jan 2012 18:46:27 EET
+ from: 1326041177 //Sun, 8 Jan 2012 18:46:27 EET
algorithms:
- algorithm:
name: default
\subsection{Billing}
-As common to most similar systems, billing in Aquarium is the application of
-a billing contract to an incoming billing event in order to produce an
-entry for the user's wallet. However, in stark contrast to most other systems,
-which rely on database transactions in order to securely modify the user's
-balance, Aquarium performs account updates asynchronously and concurrently
-for all known users.
+Commonly to most similar systems, billing in Aquarium is the application of the
+provisions of a user's contract to an incoming billing event in order to
+produce an entry for the user's wallet. However, in stark contrast to most
+other systems, which rely on database transactions in order to securely modify
+the user's balance, Aquarium performs account updates asynchronously and
+concurrently for all users.
+
+
+
+Per resource, the charging operation is affected by the cost policy and complexity
+parameters. Specifically, the 3 available cost policies affect the calculation
+of the amount of resource usage to be charged as follows:
+
+\begin{itemize}
+ \item resources employing the \textsf{continuous} cost policy are charged for
+ the actual resource usage through time. When a resource event arrives,
+ the previous resource state between the previous charge operation and the
+ current event event timestamp is charged and the resource state is then
+ updated. More formally, for continuous resources, if $f(t)$ represents
+ the function of resource usage through time and $p(t)$ is the function
+ representing the pricelist at time $t$,
+ then the total cost up to a
+ $c(t) = \sum_{i=0}^{t} {p(t) \times \int_0^{t}{f(t)dt}}$. Most resources
+ in Aquarium are continuous, for example bandwidth and disk space.
+
+ \item resources employing the \textsf{onoff} cost policy can be in two states:
+ either switched on and actively used or switched off. Therefore, the unit
+ of resource usage is time and not the actual resource usage, while the
+ period of charging is calculated only when the resource is switched on.
+ Virtual machines are examples of resources with the \textsf{onoff} cost
+ policy.
+
+ \item resources using the \textsf{distinct} cost policy are charged
+ upon usage, without time playing a role in the charge. Such resources
+ are useful for one off charges, such as the allocation of
+ virtual machine or the migration of a virtual machine to a less busy
+ host.
+
+\end{itemize}
+
+Billing events are obtained through a connection to a message queue. Upon
+arrival, a billing event is stored in an immutable log, and then forwarded to
+the user actor's mailbox; the calculation of the actual billing entries to be
+stored in the user's wallet is done within the context of the user actor,
+serially for each incoming events. This permits the actor to have mutable state
+internally (as described in Section~\ref{sec:ustate}), without risking the
+calculation correctness. The calculation process involves steps such as
+validating the resource event, resolving the current state of resource affected
+by the incoming resource event, deciding the value applicable pricelist and
+algorithm, generating entries for the user's wallet and updating the current
+resource state for the user. A significant source of complexity in the process
+is the support for temporal overriding for pricelists and algorithms: within
+the timeframe between resource updates, several policies or algorithms may be
+active. The billing algorithm must therefore split the billing period to pieces
+according the applicability of each policy/algorithm and make sure that at
+least a baseline policy is in effect in order to perform the calculation.
+Consequently, a resource event might lead to several entries to the user's wallet.
-Billing events are obtained by a connection to a reliable message queue.
-The billing event format depends on the
The actual format of the event is presented in Figure~\ref{fig:resevt}.
\begin{figure}
\end{figure}
+\subsection{User State}
+\label{sec:ustate}
+
+\section{Architecture}
+\input{arch}
\section{Performance}
first created a configurable number of users and then produced billing events
that
+
+All measurements were done using the first working version of the
+Aquarium deployment, so no real optimisation effort did take place.
+
+
+\section{Lessons Learned}
+
+One of the topics of debate while designing Aquarium was the choice of
+programming platform to use. With all user facing systems in the Okeanos cloud
+being developed in Python and the initial Aquarium designers being beginner
+Scala users (but experts in Java), the choice certainly involved risk that
+management was initially reluctant to take. However, by breaking down the
+requirements and considering the various safeguards that the software would
+need to employ in order to satisfy them, it became clear that a
+typesafe language was a hard requirement. Of the platforms examined, the {\sc
+jvm} had the richest collection of ready made components; the Akka library was
+particularly enticing for the scalability and distribution possibilities it
+offered.
+
+The choice of Scala at the moment it had been made was a high risk/high gain
+bet for GRNet. However, the development team's experience has been generally
+positive. Scala as a language was an enabling factor; case classes permitted
+the expression of data models, including the configuration {\sc dsl}, that
+could be easily be serialized or read back from wire formats while also
+promoting immutability through the use of the \texttt{copy()} constructor. The
+pervasive use of immutability allowed us to write strict, yet simple and
+concise unit tests, as the number of cases to be examined was generally low.
+The \textsf{Maybe}\footnote{\textsf{Maybe} works like \textsf{Option}, but it
+has an extra possible state (\textsf{Failed}), which allows exceptions to be
+encapsulated in the return type of a function, and then retrieved and accounted
+for in a pattern matching operation with no side effects. More at
+\url{https://github.com/loverdos/Maybe}} monad, enabled side-effect free
+development of data processing functions, even in cases where exceptions were
+the only way to go. Java interoperability was excellent, while thin Scala
+wrappers around existing Java libraries enabled higher productivity and use of
+Scala idioms in conjunction with Java code.
+
+The Akka library, which is the backbone of our system, is a prime example of
+the simplicity that can be achieved by using carefully designed high-level
+components. Akka's custom supervision hierarchies allowed us to partition the
+system in self-healing sub-components, each of which can fail independently
+of the other. For example, if the queue reader component fails due to a queue
+failure, Aquarium will still be accessible and responsive for the {\sc rest}
+interface. Also, Akka allowed us to easily saturate the processing components
+of any system we tested Aquarium on, simply by tuning the number of threads (in
+{\sc i/o} bound parts) and actors (in {\sc cpu} bound parts) per dispatcher.
+
+Despite the above, the experience was not as smooth as initially expected. The
+most prominent problem we encountered was that of lacking documentation. The
+Akka library documentation, extensive as is, only scratches the surface.
+Several other libraries we use, for example Spray for {\sc rest} handling, have
+non-existent documentation. The Java platform, and .Net that followed, has
+shown that thorough and precise documentation are key to adoption, and we
+expected a similar quality level. Related is the problem of shared community
+wisdom; as most developers know, a search for any programming problem will
+reveal several straightforward Java or scripting language sources. The
+situation with Scala is usually the opposite; the expressive power of the
+language makes it the current language of choice for treating esoteric
+functional programming concepts, while simple topics are often neglected. Scala
+has several libraries of algebraic datatypes but no {\sc yaml} parser. As Scala
+gains mainstream adoption, we hope that such problems will fade.
+
+From a software engineering point of view, the current state of the project was
+reached using about 6 person months of effort, 2 of which were devoted to
+requirements elicitation, prototype building and familiarizing with the
+language. The source code currently consists of 5.000 lines of executable
+statements (including about 1.000 lines of tests), divided in about 10
+packages. The system is built using both {\sc sbt} and Maven.
+
\section{Related Work}
\section{Conclusions and Future Work}
-In this paper, we presented Aquarium, a high-performance, generic accounting
-system, currently tuned for cloud applications.
+In this paper, we presented Aquarium, a high-performance, generic billing
+system, currently tuned for cloud applications. We presented the requirements
+that underpinned its design, outlined the architectural decisions made
+and analysed its implementation and performance.
+
+Scala has been an enabling factor for the implementation of Aquarium, both
+at the system prototyping phase and during actual development.
-Aquarium is currently under development, with a first operational version
-being planned for early 2012. All subsystems are operational, and the system
-can already support
+Aquarium is still under development, with a first stable version
+being planned for early 2012.
Aquarium is available under an open source license at
\url{https://code.grnet.gr/projects/aquarium}.
projects / aquarium / blobdiff