1 \documentclass[preprint,10pt]{sigplanconf}
5 \usepackage[british]{babel}
10 \newcommand{\cL}{{\cal L}}
11 \newcommand{\TODO}{{\sl TODO \marginpar{\sl TODO}}}
14 \conferenceinfo{ScalaDays '12}{London, UK.}
16 \copyrightdata{1-59593-056-6/05/0006}
18 \titlebanner{DRAFT---Do not distribute}
22 \title{Aquarium: Billing for the Cloud in the Cloud}
24 \authorinfo{Georgios Gousios \and Christos KK Loverdos \and Nectarios Koziris}
26 {\{gousiosg,loverdos,nkoziris\}@grnet.gr}
30 This paper describes the architecture for the Aquarium cloud infrastructure
34 \category{D.3.3}{Programming Languages}{Language Constructs and Features}[Control structures]
37 Object-Oriented Programming, Philosophy
40 OOP, Ontology, Programming Philosophy
42 \section{Introduction}
43 \section{Requirements}
45 Aquarium was designed on a clean sheet to serve a particular purpose,
46 namely the provision of billing services to an IaaS infrastructure,
47 and also be extensible to new services. In the following sections,
48 we briefly present the requirements that shaped Aquarium's design.
50 \subsection{Application Environment}
51 Aquarium developed as part of the Okeanos project at GRNet. The
52 Okeanos project is building a full stack public IaaS system for Greek
53 universities, and several services on top of it. Several components comprise
54 the Okeanos infrastructure:
58 \item[Synnefo] is an IaaS management console. Users can create and start
59 VMs, monitor their usage, create private internal networks among VMs
60 and connect to them over the web. The service backend is based on
61 Google's Ganneti for VM host management and hundrends of physical
64 \item[Archipelago] is a storage service, based on the Rados
65 distributed object store. It is currently under development, and the
66 plan is to act as the single point of storage for VM images, shared
67 volumes and user files, providing clonable snapshots and distributed
70 \item[Pithos] is a user oriented file storage service. Currently it its
71 second incarnation, it supports content deduplication, sharing of files
72 and folders and a multitude of clients.
74 \item[Astakos] is an identity consolidation system that also acts as the
75 entry point to the entire infrastructure. Users can login using
76 identities from multiple systems, such as the Shibboleth (SAML)
77 federation enabled across all Greek universities or their Twitter
82 While all the above systems (and several prospective ones) have different
83 user interfaces and provide distinct functionality in the context of
84 the GRnet IaaS, they all share a common notion of \emph{resources}, access
85 and manipulation options to which they offer to users.
89 The Okeanos IaaS will support the Greek higher education, an estimated
90 population of 100.000 students and researchers. Each member will be granted
91 access to a collection of resources using her institutional account. To enforce
92 a limit to the resources that can be acquired from the platform, each user will
93 have a limited amount of credits, renewable each month, which will be allowed
94 to spend on any resource available through the infrastructure. Resources can
95 also be shared; for example files on the Pithos file service or virtual machine
96 images on the Archipelago storage are potentially subject to concurrent usage from
97 multiple users. This means that charges for the use of a single resource
98 may need to be distributed among several users. Also this may mean that in order
99 for sharing to work correctly, users may need to transfer credits among them.
102 \subsection{Configuration}
104 Billing systems are by nature open ended. As new services are deployed, new
105 resources appear, while others might be phased out. Moreover, changes to
106 company policies may trigger changes to price lists for those resources, while
107 ad-hoc requests for large scale computational resources may require special
108 pricing policies. In order for a billing system to be able to successfully
109 adapt to changing requirements, it must be able to accommodate such changes
110 without requiring changes to the application itself. This means that all
111 information required for Aquarium in order to perform a billing operation,
112 must be provided to it externally. Moreover, to ensure high-availability,
113 billing configuration should be updatable while Aquarium is running, or at
114 least with minimal downtime, without affecting the operation of external
120 In the context of the Okeanos system, Aquarium provides billing services on a
121 per user basis for all resources exposed by other systems. As such, it is in
122 the critical path of user requests that modify resource state; all supported
123 applications must query Aquarium in order to ensure that the user has enough
124 credits to create a new resource. This means that for a large number of users
125 (given previous GRNet systems usage by the Greek research community, we
126 estimate a concurrency level of 30.000 users), Aquarium must update and
127 maintain in a queryable form their credit status,
128 with soft realtime guarantees.
130 Being on the critical path also means that Aquarium must be highly resilient,
131 too. If Aquarium fails, all supported systems will also fail. Even if Aquarium
132 fails for a short period of time, it must not loose any billing events, as this
133 will allow users to use resources without paying for them. Moreover, in case of
134 failure, Aquarium must not corrupt any billing data under any circumstances,
135 while it should reach an operating state very fast after a service restart.
137 \section{Architecture}
141 \section{Implementation}
144 \subsection{The configuration DSL}
146 The configuration requirements presented above were addressed by creating a new
147 domain specific language ({\sc dsl}), based on the YAML format. The DSL
148 enables administrators to specify billable resources, billing policies and
149 price lists and combine them arbitrarily into agreements applicable to specific
150 users, user groups or the whole system.
151 The DSL supports inheritance for policies, price lists and agreements and composition in the case of agreements.
152 It also facilitates the
153 definition of generic, repeatable debiting rules, which are then used by the
154 system to refill the user's account with credits on a periodic based.
156 The DSL is in itself based on five top-level entities, namely:
160 \item[Resources] specify the properties of resources that Aquarium knows
161 about. Apart from the expected ones (name, unit etc), a resource has
162 two properties that affect billing: \textsf{costpolicy} defines the
163 algorithm to be used to calculate the resource usage, while the
164 \textsf{complex} attribute defines whether a resource can have one or
165 many instances per user.
167 \item[Pricelists] assign a price tag to each resource, within a timeframe.
169 \item[Algorithms] specify the way the billing operation is done in response
170 to a billing event. The simplest (and default) way is to multiply the
171 billable quantity with the applicable price. To enable more complex billing
172 scenarios, the Aquarium DSL supports a simple imperative language with
173 a number of implicit variables (e.g. \texttt{price, volume, date})
174 that enable administrators to specify, e.g. billing algorithms that
175 scale with billable volume. Similarily to pricelists, algorithms
176 have an applicability timeframe attached to them.
178 \item[Crediplans] define a number of credits to give to users and a repetition
181 \item[Agreements] assign a name to algorithm, pricelist and creditplan triplets,
182 which is then assigned to each user.
188 \lstset{language=c, basicstyle=\footnotesize,
189 stringstyle=\ttfamily,
190 flexiblecolumns=true, aboveskip=-0.9em, belowskip=0em, lineskip=0em}
198 costpolicy: continuous
211 - start: "00 02 * * Tue"
213 from: 1326041177 //Sun, 8 Jan 2012 18:46:27 EET
217 bandwidthup: $price times $volume
222 name: scaledbandwidth
226 if $volume gt 15 then
228 elsif $volume gt 15 and volume lt 30 then
229 $volume times $price times 1.2
231 $volume times price times 1.4
235 \caption{A simple billing policy definition.}
239 In Figure~\ref{fig:dsl}, we present the definition of a simple (albeit valid)
240 policy. The policy parsing is done top down, so the order of definition
241 is important. The definition starts with a resource, whose name is then
242 re-used in order to attach a pricelist and a price calculation algorith to it.
243 In the case of pricelists, we present an example of \emph{temporal overloading};
244 the \texttt{everyTue2} pricelist overrides the default one, but only for
245 all repeating time frames between every Tuesday at 02:00 and Wednesday at
246 02:00, starting from the timestamp indicated at the \texttt{from} field. Another
247 example of overloading is presented at the definition of the agreement, which
248 overloads the default algorithm definition using the imperative part of the
249 Aquarium {\sc dsl} to provide a scaling charge algorithm.
253 Commonly to most similar systems, billing in Aquarium is the application of the
254 provisions of a user's contract to an incoming billing event in order to
255 produce an entry for the user's wallet. However, in stark contrast to most
256 other systems, which rely on database transactions in order to securely modify
257 the user's balance, Aquarium performs account updates asynchronously and
258 concurrently for all users.
260 Before delving into the details of what things Aquarium computes and how these computations are performed, some terminology is needed. The following are the basic entities involved.
264 The analog of money. Credits are the `universal money` within Aquarium.
267 A billable/chargeable entity. We generally need credits to use a resource. When a resource is used, then consume credits. Examples of resources are the `download bandwidth` and, respectively, the 'upload bandwidth', the `disk space` and the `VM time` to name a few. Generally speaking, the ``resource'' term specifies the type.
269 \item[Resource instance]
270 A user may have several instances of a resource type. For example, regarding a ``Virtual Machine'' resource, a user may have more then one of them. They are distinguished by their unique resource instance identifier.
272 \item[Resource instance state]
273 A value which is associated with a resource instance. Usually a floating point number, as in the $10.5$ MB designation regarding a possible ``total current downloading bandwidth''.
276 An owner of resources and credits. Users are defined externally of Aquarium.
278 \item[Resource event]
279 An event that is generated by a system, which is responsible for the resource.
280 The resource event describes a state change for the resource. In particular, a resource event records the time when that state change occurred (`occurredMillis` attribute) and the changed value (`value` attribute).
282 \item[Resource event store]
283 A datatabase, in the general sense, where resource events are stored.
286 A cost policy refers to a resource and it is the policy used in order to charge credits for resource usage.
289 A, usually, periodic statement of how many credits the user has consumed. It may contain detailed analysis that relates consumed credits to resources.
291 \item[Billing period]
292 A time period at the end of which we issue a user bill.
293 A billing period is made of a starting date and a duration that is a multiple of a week.
294 A usual billing period starts on a particular month date (eg. 3rd) and lasts for a month.
295 Each resource type designates what happens to its accumulated value (if any) at the beginning of the billing period. Usually, at the beginning of the billing period, the accumulating amounts of resources are set accumulating amount. For example, for a monthly billing period, the total uploading bandwidth is reset to zero every month.
298 The user state is made of the following distinct parts, of which the first two can be integrated to a unifying ``resource'' concept.
301 \item User credit state, that is the total credit amount for the user.
303 \item User resource state, which refers to the state of each resource instance that the user owns.
305 \item Processing state \TODO
308 \item[Resource event processing]
309 The set of algorithmic steps by which a resource event leads to state changes of the user state.
323 Per resource, the charging operation is affected by the cost policy and complexity
324 parameters. Specifically, the 3 available cost policies affect the calculation
325 of the amount of resource usage to be charged as follows:
328 \item resources employing the \textsf{continuous} cost policy are charged for
329 the actual resource usage through time. When a resource event arrives,
330 the previous resource state between the previous charge operation and the
331 current event event timestamp is charged and the resource state is then
332 updated. More formally, for continuous resources, if $f(t)$ represents
333 the function of resource usage through time and $p(t)$ is the function
334 representing the pricelist at time $t$,
335 then the total cost up to a
336 $c(t) = \sum_{i=0}^{t} {p(t) \times \int_0^{t}{f(t)dt}}$. Most resources
337 in Aquarium are continuous, for example bandwidth and disk space.
339 \item resources employing the \textsf{onoff} cost policy can be in two states:
340 either switched on and actively used or switched off. Therefore, the unit
341 of resource usage is time and not the actual resource usage, while the
342 period of charging is calculated only when the resource is switched on.
343 Virtual machines are examples of resources with the \textsf{onoff} cost
346 \item resources using the \textsf{distinct} cost policy are charged
347 upon usage, without time playing a role in the charge. Such resources
348 are useful for one off charges, such as the allocation of
349 virtual machine or the migration of a virtual machine to a less busy
354 Billing events are obtained through a connection to a message queue. Upon
355 arrival, a billing event is stored in an immutable log, and then forwarded to
356 the user actor's mailbox; the calculation of the actual billing entries to be
357 stored in the user's wallet is done within the context of the user actor,
358 serially for each incoming events. This permits the actor to have mutable state
359 internally (as described in Section~\ref{sec:ustate}), without risking the
360 calculation correctness. The calculation process involves steps such as
361 validating the resource event, resolving the current state of resource affected
362 by the incoming resource event, deciding the value applicable pricelist and
363 algorithm, generating entries for the user's wallet and updating the current
364 resource state for the user. A significant source of complexity in the process
365 is the support for temporal overriding for pricelists and algorithms: within
366 the timeframe between resource updates, several policies or algorithms may be
367 active. The billing algorithm must therefore split the billing period to pieces
368 according the applicability of each policy/algorithm and make sure that at
369 least a baseline policy is in effect in order to perform the calculation.
370 Consequently, a resource event might lead to several entries to the user's wallet.
372 The actual format of the event is presented in Figure~\ref{fig:resevt}.
375 \lstset{language=C, basicstyle=\footnotesize,
376 stringstyle=\ttfamily,
377 flexiblecolumns=true, aboveskip=-0.9em, belowskip=0em, lineskip=0em}
381 "id":"4b3288b57e5c1b08a67147c495e54a68655fdab8",
382 "occured":1314829876295,
394 \caption{A billing event example}
399 \subsection{User State}
402 \section{Performance}
404 To evaluate the performance and scalability of Aquarium, we performed two
405 experiments: The first one is a micro-benchmark that measures the time required
406 for the basic processing operation performed by Aquarium, which is billing for
407 increasing number of messages. The second one demonstrates Aquarium's
408 scalability on a single node with respect to the number of users. In both
409 cases, Aquarium was run on a MacBookPro featuring a quad core 2.33{\sc g}hz
410 Intel i7 processor and 8{\sc gb} of {\sc ram}. We selected Rabbit{\sc mq} and
411 Mongo{\sc db} as the queue and database servers, both of which were run on a
412 virtualised 4 core with 4{\sc gb} {\sc ram} Debian Linux server. Both systems
413 were run using current versions at the time of benchmarking (2.7.1 for
414 Rabbit{\sc mq} and 2.6 for Mongo{\sc db}). The two systems were connected with
415 a full duplex 100Mbps connection. No particular optimization was performed on
416 either back-end system, nor to the {\sc jvm} that run Aquarium.
418 To simulate a realistic deployment, Aquarium was configured, using the policy
419 {\sc dsl} to handle billing events for 4 types of resources, using 3 overloaded
420 pricelists, 2 overloaded algorithms, all of which were combined to 10 different
421 agreements, which were randomly (uniformly) assigned to users. To drive the
422 benchmarks, we used a synthetic load generator that worked in two stages: it
423 first created a configurable number of users and then produced billing events
427 All measurements were done using the first working version of the
428 Aquarium deployment, so no real optimisation effort did take place.
431 \section{Lessons Learned}
433 One of the topics of debate while designing Aquarium was the choice of
434 programming platform to use. With all user facing systems in the Okeanos cloud
435 being developed in Python and the initial Aquarium designers being beginner
436 Scala users (but experts in Java), the choice certainly involved risk that
437 management was initially reluctant to take. However, by breaking down the
438 requirements and considering the various safeguards that the software would
439 need to employ in order to satisfy them, it became clear that a
440 typesafe language was a hard requirement. Of the platforms examined, the {\sc
441 jvm} had the richest collection of ready made components; the Akka library was
442 particularly enticing for the scalability and distribution possibilities it
445 The choice of Scala at the moment it had been made was a high risk/high gain
446 bet for GRNet. However, the development team's experience has been generally
447 positive. Scala as a language was an enabling factor; case classes permitted
448 the expression of data models, including the configuration {\sc dsl}, that
449 could be easily be serialized or read back from wire formats while also
450 promoting immutability through the use of the \texttt{copy()} constructor. The
451 pervasive use of immutability allowed us to write strict, yet simple and
452 concise unit tests, as the number of cases to be examined was generally low.
453 The \textsf{Maybe}\footnote{\textsf{Maybe} works like \textsf{Option}, but it
454 has an extra possible state (\textsf{Failed}), which allows exceptions to be
455 encapsulated in the return type of a function, and then retrieved and accounted
456 for in a pattern matching operation with no side effects. More at
457 \url{https://github.com/loverdos/Maybe}} monad, enabled side-effect free
458 development of data processing functions, even in cases where exceptions were
459 the only way to go. Java interoperability was excellent, while thin Scala
460 wrappers around existing Java libraries enabled higher productivity and use of
461 Scala idioms in conjunction with Java code.
463 The Akka library, which is the backbone of our system, is a prime example of
464 the simplicity that can be achieved by using carefully designed high-level
465 components. Akka's custom supervision hierarchies allowed us to partition the
466 system in self-healing sub-components, each of which can fail independently
467 of the other. For example, if the queue reader component fails due to a queue
468 failure, Aquarium will still be accessible and responsive for the {\sc rest}
469 interface. Also, Akka allowed us to easily saturate the processing components
470 of any system we tested Aquarium on, simply by tuning the number of threads (in
471 {\sc i/o} bound parts) and actors (in {\sc cpu} bound parts) per dispatcher.
473 Despite the above, the experience was not as smooth as initially expected. The
474 most prominent problem we encountered was that of lacking documentation. The
475 Akka library documentation, extensive as is, only scratches the surface.
476 Several other libraries we use, for example Spray for {\sc rest} handling, have
477 non-existent documentation. The Java platform, and .Net that followed, has
478 shown that thorough and precise documentation are key to adoption, and we
479 expected a similar quality level. Related is the problem of shared community
480 wisdom; as most developers know, a search for any programming problem will
481 reveal several straightforward Java or scripting language sources. The
482 situation with Scala is usually the opposite; the expressive power of the
483 language makes it the current language of choice for treating esoteric
484 functional programming concepts, while simple topics are often neglected. Scala
485 has several libraries of algebraic datatypes but no {\sc yaml} parser. As Scala
486 gains mainstream adoption, we hope that such problems will fade.
488 From a software engineering point of view, the current state of the project was
489 reached using about 6 person months of effort, 2 of which were devoted to
490 requirements elicitation, prototype building and familiarizing with the
491 language. The source code currently consists of 5.000 lines of executable
492 statements (including about 1.000 lines of tests), divided in about 10
493 packages. The system is built using both {\sc sbt} and Maven.
495 \section{Related Work}
497 \section{Conclusions and Future Work}
498 In this paper, we presented Aquarium, a high-performance, generic billing
499 system, currently tuned for cloud applications. We presented the requirements
500 that underpinned its design, outlined the architectural decisions made
501 and analysed its implementation and performance.
503 Scala has been an enabling factor for the implementation of Aquarium, both
504 at the system prototyping phase and during actual development.
506 Aquarium is still under development, with a first stable version
507 being planned for early 2012.
509 Aquarium is available under an open source license at
510 \url{https://code.grnet.gr/projects/aquarium}.
512 \bibliographystyle{abbrvnat}
513 \bibliography{aquarium}