A great contribution to the paper.

[aquarium] / doc / arch / aquarium.tex

\documentclass[preprint,10pt]{sigplanconf}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{graphicx}
\usepackage[british]{babel}
\usepackage{url}
\usepackage{listings}
\usepackage{color}

\newcommand{\cL}{{\cal L}}

\begin{document}
\conferenceinfo{ScalaDays '12}{London, UK.}
\copyrightyear{2012}
\copyrightdata{1-59593-056-6/05/0006}

\titlebanner{DRAFT---Do not distribute}



\title{Aquarium: Billing for the Cloud in the Cloud}

\authorinfo{Georgios Gousios \and Christos KK Loverdos}
{GRNet SA}
{\{gousiosg,loverdos\}@grnet.gr}

\maketitle
\begin{abstract}
    This paper describes the architecture for the Aquarium cloud infrastructure
    software.
\end{abstract}

\category{D.3.3}{Programming Languages}{Language Constructs and Features}[Control structures]

\terms
    Object-Oriented Programming, Philosophy

\keywords
    OOP, Ontology, Programming Philosophy

\section{Introduction}
\section{Requirements}

Aquarium was designed on a clean sheet to serve a particular purpose,
namely the provision of billing services to an IaaS infrastructure,
and also be extensible to new services. In the following sections,
we briefly present the requirements that shaped Aquarium's design.

\subsection{Application Environment}
Aquarium developed as part of the Okeanos project at GRNet. The
Okeanos project is building a full stack public IaaS system for Greek
universities, and several services on top of it. Several components comprise
the Okeanos infrastructure:

\begin{description}

    \item[Synnefo] is an IaaS management console. Users can create and start
        VMs, monitor their usage, create private internal networks among VMs
        and connect to them over the web. The service backend is based on
        Google's Ganneti for VM host management and hundrends of physical
        VM container nodes.

    \item[Archipelago] is a storage service, based on the Rados
        distributed object store. It is currently under development, and the
        plan is to act as the single point of storage for VM images, shared
        volumes and user files, providing clonable snapshots and distributed
        fault tolerance.
    
    \item[Pithos] is a user oriented file storage service. Currently it its
        second incarnation, it supports content deduplication, sharing of files
        and folders and a multitude of clients.

    \item[Astakos] is an identity consolidation system that also acts as the
        entry point to the entire infrastructure. Users can login using 
        identities from multiple systems, such as the Shibboleth (SAML) 
        federation enabled across all Greek universities or their Twitter 
        accounts.

\end{description}

While all the above systems (and several prospective ones) have different 
user interfaces and provide distinct functionality in the context of
the GRnet IaaS, they all share a common notion of \emph{resources}, access
and manipulation options to which they offer to users. 

\subsection{Supported Users}

The Okeanos IaaS will support the Greek higher education, an estimated
population of 100.000 students and researchers. Each member will be granted
access to a collection of resources using her institutional account. To enforce
a limit to the resources that can be acquired from the platform, each user will
have a limited amount of credits, renewable each month, which will be allowed
to spend on any resource available through the infrastructure. Resources can
also be shared; for example files on the Pithos file service or virtual machine
images on the Archipelago storage can be 


\subsection{Configuration}

Billing systems are by nature open ended. As new services are deployed, new
resources appear, while others might be phased out.  Moreover, changes to
company policies may trigger changes to price lists for those resources, while
ad-hoc requests for large scale computational resources may require special
pricing policies. In order for a billing system to be able to successfully
adapt to changing requirements, it must be able to accommodate such changes
without requiring changes to the application itself. This means that all
information required for Aquarium in order to perform a billing operation,
must be provided to it externally. Moreover, to ensure high-availability,
billing configuration should be updatable while Aquarium is running, or at
least with minimal downtime, without affecting the operation of external
systems.


\subsection{Scaling}

In the context of the Okeanos system, Aquarium provides billing services on a
per user basis for all resources exposed by other systems. As such, it is in
the critical path of user requests that modify resource state; all supported
applications must query Aquarium in order to ensure that the user has enough
credits to create a new resource. This means that for a large number of users
(given previous GRNet systems usage by the Greek research community, we
estimate a concurrency level of 30.000 users), Aquarium must update and
maintain in a queryable form their credit status, 
with soft realtime guarantees. 

Being on the critical path also means that Aquarium must be highly resilient,
too. If Aquarium fails, all supported systems will also fail. Even if Aquarium
fails for a short period of time, it must not loose any billing events, as this
will allow users to use resources without paying for them. Moreover, in case of
failure, Aquarium must not corrupt any billing data under any circumstances,
while it should reach an operating state very fast after a service restart.

\section{Architecture}



\section{Implementation}

\subsection{The configuration DSL}

The configuration requirements presented above were addressed by creating a new
domain specific language ({\sc dsl}), based on the YAML format.  The DSL
enables administrators to specify billable resources, billing policies and
price lists and combine them arbitrarily into agreements applicable to specific
users, user groups or the whole system. 
The DSL supports inheritance for policies, price lists and agreements and composition in the case of agreements.
It also facilitates the
definition of generic, repeatable debiting rules, which are then used by the
system to refill the user's account with credits on a periodic based.

The DSL is in itself based on five top-level entities, namely:

\begin{description}

    \item[Resources] specify the properties of resources that Aquarium knows
        about. Apart from the expected ones (name, unit etc), 
        a resource has two properties that affect billing: \textsf{costpolicy}
        defines whether the billing operation is to be performed at the moment
        a billing event has arrived, while the \textsf{complex} attribute defines
        whether a resource can have many instances per user.

    \item[Pricelists] assign a price tag to each resource, within a timeframe.
    
    \item[Algorithms] specify the way the billing operation is done in response
        to a billing event. The simplest (and default) way is to multiply the 
        billable quantity with the applicable price. To enable more complex billing
        scenarios, the Aquarium DSL supports a simple imperative language with
        a number of implicit variables (e.g. \texttt{price, volume, date}) 
        that enable administrators to specify, e.g. billing algorithms that
        scale with billable volume. Similarily to pricelists, algorithms
        have an applicability timeframe attached to them.

    \item[Crediplans] define a number of credits to give to users and a repetition
        period.

    \item[Agreements] assign a name to algorithm, pricelist and creditplan triplets,
        which is then assigned to each user.

\end{description}


\begin{figure}
\lstset{language=ruby, basicstyle=\footnotesize,
stringstyle=\ttfamily, 
flexiblecolumns=true, aboveskip=-0.9em, belowskip=0em, lineskip=0em}

\begin{lstlisting}
resources:
  - resource:
    name: bandwidthup
    unit: MB/hr
    complex: false
    costpolicy: continuous
pricelists:
  - pricelist: 
    name: default
    bandwidthup: 0.01
    effective:
      from: 0
  - pricelist: 
    name: everyTue2
    overrides: default
    bandwidthup: 0.1
    effective:
      repeat:
      - start: "00 02 * * Tue"
        end:   "00 02 * * Wed"
      from: 1326041177 #Sun, 8 Jan 2012 18:46:27 EET
algorithms:
  - algorithm:
    name: default
    bandwidthup: $price times $volume
    effective:
      from: 0
agreements:
  - agreement:
    name: scaledbandwidth
    pricelist: everyTue2
    algorithm:
      bandwidthup: |
        if $volume gt 15 then
          $volume times $price
        elsif $volume gt 15 and volume lt 30 then
          $volume times $price times 1.2
        else
          $volume times price times 1.4
        end
\end{lstlisting}

\caption{A simple billing policy definition.} 
\label{fig:dsl}
\end{figure}

In Figure~\ref{fig:dsl}, we present the definition of a simple (albeit valid) 
policy. The policy parsing is done top down, so the order of definition 
is important. The definition starts with a resource, whose name is then
re-used in order to attach a pricelist and a price calculation algorith to it.
In the case of pricelists, we present an example of \emph{temporal overloading};
the \texttt{everyTue2} pricelist overrides the default one, but only for 
all repeating time frames between every Tuesday at 02:00 and Wednesday at
02:00, starting from the timestamp indicated at the \texttt{from} field. Another
example of overloading is presented at the definition of the agreement, which
overloads the default algorithm definition using the imperative part of the
Aquarium {\sc dsl} to provide a scaling charge algorithm.

\subsection{Billing}

As common to most similar systems, billing in Aquarium is the application of
a billing contract to an incoming billing event in order to produce an 
entry for the user's wallet. However, in stark contrast to most other systems,
which rely on database transactions in order to securely modify the user's
balance, Aquarium performs account updates asynchronously and concurrently
for all known users.

Billing events are obtained by a connection to a reliable message queue.
The billing event format depends on the 
The actual format of the event is presented in Figure~\ref{fig:resevt}.

\begin{figure}
\lstset{language=C, basicstyle=\footnotesize,
stringstyle=\ttfamily, 
flexiblecolumns=true, aboveskip=-0.9em, belowskip=0em, lineskip=0em}

\begin{lstlisting}
{
  "id":"4b3288b57e5c1b08a67147c495e54a68655fdab8",
  "occured":1314829876295,
  "userId":31,
  "cliendId":3,
  "resource":"vmtime",
  "eventVersion":1,
  "value": 1,
  "details":{
    "vmid":"3300",
    "action": "on"
  }
}
\end{lstlisting}
\caption{A billing event example} 
\label{fig:resevt}

\end{figure}

\subsection{Implementation Experience}

One of the topics of debate while designing Aquarium was the choice of
programming platform to use. With all user facing systems in the Okeanos cloud
being developed in Python and the initial Aquarium designers being beginner
Scala users (but experts in Java), the choice certainly involved risk that
management was initially reluctant to take. However, by breaking down the
requirements and considering the various safeguards that the software would
need to employ in order to satisfy them, it became clear that a
typesafe language was a hard requirement. Of the platforms examined, the {\sc
jvm} had the richest collection of ready made components; the Akka library was
particularly enticing for its scalability and distribution possibilities it
offered.

The choice of Scala at the moment was a high risk/high gain bet for GRNet.
However, the development team's experience has been generally positive.  Scala
as a language was an enabling factor; case classes permitted the expression of
data models, including the configuration {\sc dsl}, that could be easily be
serialized or read back from wire formats while also promoting immutability
through the use of the \texttt{copy()} constructor.  The very active use of
immutability allowed us to write strict, yet simple and concise unit tests, as
the number of cases to be examined was generally low. The
\textsf{Maybe}\footnote{\textsf{Maybe} works like \textsf{Option}, but it has
an extra possible state (\textsf{Failed}), which allows exceptions to be
encapsulated in the return type of a function, and then retrieved and accounted
for in a pattern matching operation with no side effects. More at
\url{https://github.com/loverdos/Maybe}} monad, developed by the second author,
enabled side-effect free development of data processing functions, even in cases
where exceptions were the only way to go. Java interoperability was excellent,
while thin Scala wrappers around existing Java libraries enabled higher
productivity and use of Scala idioms in conjunction with Java code.

Despite the above, the experience was not as smooth as initially expected. The
most prominent problem we encountered was that of missing documentation. The
Akka library documentation, extensive as is, only scratches the surface.
Several other libraries we use, for example Spray for {\sc rest} handling, have
non-existent documentation. The Java platform, and .Net that followed, has
shown that thorough and precise documentation are key to adoption, and we
expected a similar quality level. Related is the problem of shared community
wisdom; as most developers know, a search for any programming problem will
reveal several straightforward Java or scripting language sources. The
situation with Scala is usually the opposite; the expressive power of the
language makes it the current language of choice for treating esoteric
functional programming concepts, while simple topics are often neglected. Scala
has several libraries of algebraic datatypes but no {\sc yaml} parser. As Scala
gains mainstream adoption, we hope that such problems will fade.

From a software engineering point of view, the current state of the project was
reached using about 6 person months of effort, 2 of which were devoted to
requirements elicitation, prototype building and familiarizing with the
language. The source code currently consists of 5.000 lines of executable
statements (including about 1.000 lines of tests), divided in about 10
packages. The system is built using both {\sc sbt} and Maven. 

\section{Performance}

To evaluate the performance and scalability of Aquarium, we performed two
experiments: The first one is a micro-benchmark that measures the time required
for the basic processing operation performed by Aquarium, which is billing for
increasing number of messages. The second one demonstrates Aquarium's
scalability on a single node with respect to the number of users.  In both
cases, Aquarium was run on a MacBookPro featuring a quad core 2.33{\sc g}hz
Intel i7 processor and 8{\sc gb} of {\sc ram}. We selected Rabbit{\sc mq} and
Mongo{\sc db} as the queue and database servers, both of which were run on a
virtualised 4 core with 4{\sc gb} {\sc ram} Debian Linux server. Both systems
were run using current versions at the time of benchmarking (2.7.1 for
Rabbit{\sc mq} and 2.6 for Mongo{\sc db}).  The two systems were connected with
a full duplex 100Mbps connection.  No particular optimization was performed on
either back-end system, nor to the {\sc jvm} that run Aquarium. 

To simulate a realistic deployment, Aquarium was configured, using the policy
{\sc dsl} to handle billing events for 4 types of resources, using 3 overloaded
pricelists, 2 overloaded algorithms, all of which were combined to 10 different
agreements, which were randomly (uniformly) assigned to users. To drive the
benchmarks, we used a synthetic load generator that worked in two stages: it
first created a configurable number of users and then produced billing events
that 


The measurements above were done on the first working version of the
Aquarium deployment. They present 

\section{Related Work}

\section{Conclusions and Future Work}
In this paper, we presented Aquarium, a high-performance, generic billing 
system, currently tuned for cloud applications. We presented the requirements
that underpinned its design, outlined the architectural decisions made
and analysed its implementation and performance.

Scala has been an enabling factor for the implementation of Aquarium, both
at the system prototyping phase and during actual development. 

Aquarium is still under development, with a first stable version 
being planned for early 2012. 

Aquarium is available under an open source license at 
\url{https://code.grnet.gr/projects/aquarium}.

\bibliographystyle{abbrvnat}
\bibliography{aquarium}

\end{document}