root / README.deploy @ 480051fe
History | View | Annotate | Download (19.8 kB)
1 |
README.deploy -- Instructions for a basic Synnefo deployment |
---|---|
2 |
|
3 |
This document describes the basic steps to obtain a basic, working Synnefo |
4 |
deployment. It begins by examining the different node roles, then moves to the |
5 |
installation and setup of distinct software components. |
6 |
|
7 |
It is current as of Synnefo v0.5.2. |
8 |
|
9 |
|
10 |
Node types |
11 |
=========== |
12 |
|
13 |
Nodes in a Synnefo deployment belong in one of the following types: |
14 |
|
15 |
* DB: |
16 |
A node [or more than one nodes, if using an HA configuration], running a DB |
17 |
engine supported by the Django ORM layer. The DB is the single source of |
18 |
truth for the servicing of API requests by Synnefo. |
19 |
Services: PostgreSQL / MySQL |
20 |
|
21 |
* APISERVER: |
22 |
A node running the implementation of the OpenStack API, in Django. Any number |
23 |
of APISERVERs can be used, in a load-balancing configuration, without any |
24 |
special consideration. Access to a common DB ensures consistency. |
25 |
Services: Web server, vncauthproxy |
26 |
|
27 |
* QUEUE: |
28 |
A node running the RabbitMQ software, which provides AMQP functionality. More |
29 |
than one QUEUE nodes may be deployed, in an HA configuration. Such |
30 |
deployments require shared storage, provided e.g., by DRBD. |
31 |
Services: RabbitMQ [rabbitmq-server] |
32 |
|
33 |
* LOGIC: |
34 |
A node running the business logic of Synnefo, in Django. It dequeues |
35 |
messages from QUEUE nodes, and provides the context in which business logic |
36 |
functions run. It uses Django ORM to connect to the common DB and update the |
37 |
state of the system, based on notifications received from the rest of the |
38 |
infrastructure, over AMQP. |
39 |
Services: the Synnefo logic dispatcher [/logic/dispatcher.py] |
40 |
|
41 |
* GANETI-MASTER and GANETI-NODE: |
42 |
A single GANETI-MASTER and a large number of GANETI-NODEs constitute the |
43 |
Ganeti backend for Synnefo, which undertakes all VM management functions. |
44 |
Any APISERVER can issue commands to the GANETI-MASTER, over RAPI, to effect |
45 |
changes in the state of the VMs. The GANETI-MASTER runs the Ganeti request |
46 |
queue. |
47 |
Services: |
48 |
only on GANETI-MASTER: |
49 |
the Synnefo Ganeti monitoring daemon [/ganeti/snf-ganeti-eventd] |
50 |
the Synnefo Ganeti hook [/ganeti/snf-ganeti-hook.py]. |
51 |
on each GANETI_NODE: |
52 |
a deployment-specific KVM ifup script |
53 |
properly configured NFDHCPD |
54 |
|
55 |
|
56 |
Installation Process |
57 |
===================== |
58 |
|
59 |
This section describes the installation process of the various node roles in a |
60 |
Synnefo deployment. |
61 |
|
62 |
|
63 |
0. Allocation of physical nodes: |
64 |
Determine the role of every physical node in your deployment. |
65 |
|
66 |
|
67 |
1. Ganeti installation: |
68 |
Synnefo requires a working Ganeti installation at the backend. Installation |
69 |
of Ganeti is not covered by this document, please refer to |
70 |
http://docs.ganeti.org/ganeti/current/html for all the gory details. A |
71 |
successful Ganeti installation concludes with a working GANETI-MASTER and a |
72 |
number of GANETI-NODEs. |
73 |
|
74 |
|
75 |
2. RabbitMQ installation: |
76 |
RabbitMQ is used as a generic message broker for the system. It should be |
77 |
installed on two seperate QUEUE nodes (VMs should be enough for the moment) |
78 |
in a high availability configuration as described here: |
79 |
|
80 |
http://www.rabbitmq.com/pacemaker.html |
81 |
|
82 |
After installation, create a user and set its permissions |
83 |
rabbitmqctl add_user okeanos 0k3@n0s |
84 |
rabbitmqctl set_permissions -p / okeanos "^.*" ".*" ".*" |
85 |
|
86 |
The values set for the user and password must be mirrored in the |
87 |
RABBIT_* variables in settings.py (see step 6) |
88 |
|
89 |
|
90 |
3. Web server installation: |
91 |
A Web Server (e.g., Apache) needs to be installed on the APISERVERs, |
92 |
and be configured to run the Synnefo Django project appropriately. Selection |
93 |
and configuration of a Web server is outside the scope of this document. |
94 |
|
95 |
For testing or development purposes, Django's own development server, |
96 |
`./manage.py runserver' can be used. |
97 |
|
98 |
|
99 |
4. Installation of the Synnefo Django project: |
100 |
As of v0.5 the Synnefo Django project needs to be installed on nodes |
101 |
of type APISERVER, and LOGIC, with a properly configured settings.py. In |
102 |
later revisions, the specific parts of the Django project which need to run |
103 |
on each node type will be identified. |
104 |
|
105 |
Synnefo is written in Python 2.6 and depends on the following Python modules: |
106 |
[package versions confirmed to be compatible are in braces] |
107 |
|
108 |
* django 1.2 [Django==1.2.4] |
109 |
* simplejson [simplejson==2.1.3] |
110 |
* pycurl [pycurl==7.19.0] |
111 |
* python-dateutil [python-dateutil==1.4.1] |
112 |
WARNING: version python-dateutil==2.0 downloaded by pip known *not* to |
113 |
work with Python 2.6 |
114 |
* python-ipy [IPy==0.75] |
115 |
also verified to work with python-ipy 0.70-1 as shipped with Squeeze |
116 |
* south [south==0.7.1] |
117 |
WARNING: might not work with Debian Squeeze's default south-0.7-1 package. |
118 |
* amqplib [amqplib==0.6.1] |
119 |
* lockfile [lockfile==0.8] |
120 |
* python-daemon [python-daemon==1.5.5] |
121 |
* python-prctl [python-prctl==1.3.0] |
122 |
|
123 |
also, depending on the database engine of choice, on one of the following: |
124 |
* MySQL-python [MySQL-python==1.2.3] |
125 |
* psycopg2 [psycopg2==2.4] |
126 |
|
127 |
if the invitations application is deployed, the following dependencies should |
128 |
be installed: |
129 |
* pycrypto==2.1.0 |
130 |
|
131 |
The integration test suite snf-tools/snf-test depends on: |
132 |
* python-unittest2 [unittest2==0.5.1] |
133 |
* python-paramiko [paramiko==1.7.6], version included in Debian Squeeze |
134 |
is broken wrt to use of RandomPool, see Debian bug #576697 |
135 |
* python-ipy [IPy==0.75] |
136 |
* python-prctl [python-prctl==1.3.0] |
137 |
* the client component of vncauthproxy, see Step 12 |
138 |
* the kamaki client library, please see |
139 |
https://code.grnet.gr/projects/kamaki for installation instructions. |
140 |
[FIXME: Update instructions on kamaki installation] |
141 |
|
142 |
To run the user interface tests, selenium must be installed |
143 |
* selenium [?] |
144 |
|
145 |
The easiest method for installation of the Django project is to setup a |
146 |
working environment through virtualenv. Alternatively, you can use your |
147 |
system's package manager to install the dependencies (e.g. Macports has them |
148 |
all). |
149 |
|
150 |
* On Snow Leopard and linux (64-bit), you have to set the following |
151 |
environment variable for pip to compile the dependencies correctly. |
152 |
|
153 |
$ export ARCHFLAGS="-arch x86_64" |
154 |
|
155 |
* On Ubuntu, a few more packages must be installed before installing the |
156 |
prerequisite Python libraries |
157 |
|
158 |
$ sudo aptitude install libcurl3-gnutls libcurl3-gnutls-dev uuid-dev |
159 |
|
160 |
Checkout the code and install the Python prerequisites. This assumes that |
161 |
python is already installed on the host. |
162 |
|
163 |
$ sudo easy_install virtualenv |
164 |
$ git clone https://user@code.grnet.gr/git/synnefo synnefo |
165 |
$ virtualenv --python=python2.6 synnefo --no-site-packages |
166 |
... |
167 |
$ cd synnefo |
168 |
$ ./bin/pip install <list_of_dependencies> |
169 |
|
170 |
[WARNING]: The software must be checked out in a directory named synnefo, |
171 |
otherwise python imports will not work. Therefore, do not change the |
172 |
or rename the checkout path. |
173 |
|
174 |
|
175 |
5. Database installation: |
176 |
A database supported by the Django ORM layer must be installed on nodes |
177 |
of type DB. The choices are: SQLIte, MySQL, PostgreSQL. |
178 |
|
179 |
* SQLite: |
180 |
The python sqlite driver is available by default with Python so no |
181 |
additional configuration is required. Also, most self-respecting systems |
182 |
have the sqlite library installed by default. |
183 |
|
184 |
* MySQL: |
185 |
MySQL must be installed first: |
186 |
|
187 |
* Ubuntu - Debian |
188 |
$ sudo apt-get install libmysqlclient-dev |
189 |
|
190 |
* MacPorts |
191 |
$ sudo port install mysql5 |
192 |
|
193 |
Install the MySQL python library on servers running the Django project: |
194 |
|
195 |
$ bin/pip install MySQL-python |
196 |
|
197 |
Note: On MacOSX with Mysql install from MacPorts the above command will |
198 |
fail complaining that it cannot find the mysql_config command. Do |
199 |
the following and restart the installation |
200 |
$ echo "mysql_config = /opt/local/bin/mysql_config5" >> \ |
201 |
./build/MySQL-python/site.cfg |
202 |
|
203 |
Configure a MySQL db/account for synnefo |
204 |
$ mysql -u root -p |
205 |
|
206 |
mysql> create database synnefo; |
207 |
mysql> show databases; |
208 |
mysql> GRANT ALL on synnefo.* TO username IDENTIFIED BY 'password'; |
209 |
|
210 |
IMPORTANT: |
211 |
MySQL *must* be set in READ-COMMITED mode, e.g. by setting |
212 |
|
213 |
transaction-isolation = READ-COMMITTED |
214 |
|
215 |
in the [mysqld] section of /etc/mysql/my.cnf. |
216 |
|
217 |
Alternatively, make sure the following code fragment stays enabled |
218 |
in settings.d/10-database.conf: |
219 |
|
220 |
if DATABASES['default']['ENGINE'].endswith('mysql'): |
221 |
DATABASES['default']['OPTIONS'] = { |
222 |
'init_command': 'SET storage_engine=INNODB; ' + |
223 |
'SET SESSION TRANSACTION ISOLATION LEVEL READ COMMITTED', |
224 |
} |
225 |
|
226 |
* PostgreSQL |
227 |
You need to install the PostgreSQL binaries: |
228 |
* Ubuntu - Debian |
229 |
$ sudo apt-get install postgresql-8.4 libpq-dev |
230 |
|
231 |
* MacPorts |
232 |
$ sudo port install postgresql84 |
233 |
|
234 |
Install the postgres Python library |
235 |
$ bin/pip install psycopg2 |
236 |
|
237 |
Configure a postgres db/account for synnefo: |
238 |
|
239 |
Become the postgres user, connect to PostgreSQL: |
240 |
$ sudo su - postgres |
241 |
$ psql |
242 |
|
243 |
Run the following commands: |
244 |
DROP DATABASE synnefo; |
245 |
DROP USER username; |
246 |
CREATE USER username WITH PASSWORD 'password'; |
247 |
CREATE DATABASE synnefo; |
248 |
GRANT ALL PRIVILEGES ON DATABASE synnefo TO username; |
249 |
ALTER DATABASE synnefo OWNER TO username; |
250 |
ALTER USER username CREATEDB; |
251 |
|
252 |
The last line enables the newly created user to create own databases. This |
253 |
is needed for Django to create and drop the test_synnefo database for unit |
254 |
testing. |
255 |
|
256 |
|
257 |
6. Setting up the Django project: |
258 |
The settings.py file for Django may be derived by concatenating the |
259 |
settings.py.dist file contained in the Synnefo distribution with a file |
260 |
containing custom modifications, which shall override all settings deviating |
261 |
from the supplied settings.py.dist. This is recommended to minimize the load |
262 |
of reconstructing settings.py from scratch, since each release currently |
263 |
brings heavy changes to settings.py.dist. |
264 |
|
265 |
Add the following to your custom settings.py, depending on your choice |
266 |
of DB: |
267 |
* SQLite |
268 |
|
269 |
PROJECT_PATH = os.path.dirname(os.path.abspath(__file__)) + '/' |
270 |
|
271 |
DATABASES = { |
272 |
'default': { |
273 |
'ENGINE': 'django.db.backends.sqlite3', |
274 |
'NAME': PROJECT_PATH + 'synnefo.db' # WARN: This must be an absolute path |
275 |
} |
276 |
} |
277 |
|
278 |
* MySQL |
279 |
|
280 |
DATABASES = { |
281 |
'default': { |
282 |
'ENGINE': 'django.db.backends.mysql', |
283 |
'NAME': 'synnefo', |
284 |
'USER': 'USERNAME', |
285 |
'PASSWORD': 'PASSWORD', |
286 |
'HOST': 'HOST', |
287 |
'PORT': 'PORT', |
288 |
'OPTIONS': { |
289 |
'init_command': 'SET storage_engine=INNODB', |
290 |
} |
291 |
} |
292 |
} |
293 |
|
294 |
* PostgreSQL |
295 |
|
296 |
DATABASES = { |
297 |
'default': { |
298 |
'ENGINE': 'django.db.backends.postgresql_psycopg2', |
299 |
'NAME': 'DATABASE', |
300 |
'USER': 'USERNAME', |
301 |
'PASSWORD': 'PASSWORD', |
302 |
'HOST': 'HOST', |
303 |
'PORT': 'PORT', |
304 |
} |
305 |
} |
306 |
|
307 |
Try it out. The following command will attempt to connect to the DB and |
308 |
print out DDL statements. It should not fail. |
309 |
|
310 |
$ ./bin/python manage.py sql db |
311 |
|
312 |
|
313 |
7. Initialization of Synnefo DB: |
314 |
You need to initialize the Synnefo DB and load fixtures |
315 |
db/fixtures/{users,flavors,images}.json, which make the API usable by end |
316 |
users by defining a sample set of users, hardware configurations (flavors) |
317 |
and OS images. |
318 |
|
319 |
IMPORTANT: Be sure to modify db/fixtures/users.json and select |
320 |
a unique token for each of the initial and any other users defined in this |
321 |
file. DO NOT LEAVE THE SAMPLE AUTHENTICATION TOKENS enabled in deployed |
322 |
configurations. |
323 |
|
324 |
$ ./bin/python manage.py syncdb |
325 |
$ ./bin/python manage.py migrate db |
326 |
$ ./bin/python manage.py loaddata db/fixtures/users.json |
327 |
$ ./bin/python manage.py loaddata db/fixtures/flavors.json |
328 |
$ ./bin/python manage.py loaddata db/fixtures/images.json |
329 |
|
330 |
|
331 |
8. Finalization of settings.py: |
332 |
Set the BACKEND_PREFIX_ID variable to some unique prefix, e.g. your commit |
333 |
username in settings.py. Several functional conventions within the system |
334 |
require this variable to include a dash at its end (e.g. snf-) |
335 |
|
336 |
|
337 |
9. Installation of the Ganeti monitoring daemon, /ganeti/snf-ganeti-eventd: |
338 |
The Ganeti monitoring daemon must run on GANETI-MASTER. |
339 |
|
340 |
The monitoring daemon is configured through /etc/synnefo/settings.conf. |
341 |
An example is provided under snf-ganeti-tools/. |
342 |
|
343 |
If run from the repository directory, make sure to have snf-ganeti-tools/ |
344 |
in the PYTHONPATH. |
345 |
|
346 |
You may also build Debian packages directly from the repository: |
347 |
$ cd snf-ganeti-tools |
348 |
$ dpkg-buildpackage -b -uc -us |
349 |
# dpkg -i ../snf-ganeti-tools-*deb |
350 |
|
351 |
TBD: how to handle master migration. |
352 |
|
353 |
|
354 |
10. Installation of the Synnefo dispatcher, /logic/dispatcher.py: |
355 |
The logic dispatcher is part of the Synnefo Django project and must run |
356 |
on LOGIC nodes. |
357 |
|
358 |
The dispatcher retrieves messages from the queue and calls the appropriate |
359 |
handler function as defined in the queue configuration in `setttings.py'. |
360 |
The default configuration should work directly without any modifications. |
361 |
|
362 |
For the time being The dispatcher must be run by hand: |
363 |
$ ./bin/python ./logic/dispatcher.py |
364 |
|
365 |
The dispatcher should run in at least 2 instances to ensure high |
366 |
(actually, increased) availability. |
367 |
|
368 |
|
369 |
11. Installation of the Synnefo Ganeti hook: |
370 |
The generic Synnefo Ganeti hook wrapper resides in the snf-ganeti-tools/ |
371 |
directory of the Synnefo repository. |
372 |
|
373 |
The hook needs to be enabled for phases post-{add,modify,reboot,start,stop} |
374 |
by *symlinking* in |
375 |
/etc/ganeti/hooks/instance-{add,modify,reboot,start,stop}-post.d on |
376 |
GANETI-MASTER, e.g.: |
377 |
|
378 |
root@ganeti-master:/etc/ganeti/hooks/instance-start-post.d# ls -l |
379 |
lrwxrwxrwx 1 root root 45 May 3 13:45 00-snf-ganeti-hook -> /home/devel/synnefo/snf-ganeti-hook/snf-ganeti-hook.py |
380 |
|
381 |
IMPORTANT: The link name may only contain "upper and lower case, digits, |
382 |
underscores and hyphens. In other words, the regexp ^[a-zA-Z0-9_-]+$." |
383 |
See: |
384 |
http://docs.ganeti.org/ganeti/master/html/hooks.html?highlight=hooks#naming |
385 |
|
386 |
If run from the repository directory, make sure to have snf-ganeti-tools/ |
387 |
in the PYTHONPATH. |
388 |
|
389 |
Alternative, build Debian packages which take care of building, installing |
390 |
and activating the Ganeti hook automatically, see step. 9. |
391 |
|
392 |
|
393 |
12. Installation of the VNC authentication proxy, vncauthproxy: |
394 |
To support OOB console access to the VMs over VNC, the vncauthproxy |
395 |
daemon must be running on every node of type APISERVER. |
396 |
|
397 |
Download and install vncauthproxy from its own repository, |
398 |
at https://code.grnet.gr/git/vncauthproxy (known good commit: tag v1.0). |
399 |
|
400 |
Download and install a specific repository commit: |
401 |
|
402 |
$ bin/pip install -e git+https://code.grnet.gr/git/vncauthproxy@INSERT_COMMIT_HERE#egg=vncauthproxy |
403 |
|
404 |
Create /var/log/vncauthproxy and set its permissions appropriately. |
405 |
|
406 |
Alternatively, you can build Debian packages. To do so, |
407 |
checkout the "debian" branch of the vncauthproxy repository |
408 |
(known good commit: tag debian/v1.0): |
409 |
|
410 |
$ git checkout debian |
411 |
|
412 |
Then build debian package, and install as root: |
413 |
|
414 |
$ dpkg-buildpackage -b -uc -us |
415 |
# dpkg -i ../vncauthproxy_1.0-1_all.deb |
416 |
|
417 |
--Failure to build the package on the Mac. |
418 |
|
419 |
libevent, a requirement for gevent which in turn is a requirement for |
420 |
vncauthproxy is not included in MacOSX by default and installing it with |
421 |
MacPorts does not lead to a version that can be found by the gevent |
422 |
build process. A quick workaround is to execute the following commands: |
423 |
|
424 |
cd $SYNNEFO |
425 |
sudo pip install -e git+https://code.grnet.gr/git/vncauthproxy@5a196d8481e171a#egg=vncauthproxy |
426 |
<the above fails> |
427 |
cd build/gevent |
428 |
sudo python setup.py -I/opt/local/include -L/opt/local/lib build |
429 |
cd $SYNNEFO |
430 |
sudo pip install -e git+https://code.grnet.gr/git/vncauthproxy@5a196d8481e171a#egg=vncauthproxy |
431 |
|
432 |
|
433 |
13. Installation of the customized Ganeti Instance Image for image deployment: |
434 |
For Synnefo to be able to launch VMs from specified Images, you need |
435 |
the gnt-instance-image OS Provider installed on *all* Ganeti nodes. |
436 |
There are 2 different ways to install gnt-instance-image on a node: |
437 |
|
438 |
1.As a debian package (recommended) |
439 |
|
440 |
Download and install the debian package found here: |
441 |
https://code.grnet.gr/attachments/download/351/ganeti-instance-image_0.5.1-1-snf1_all.deb |
442 |
You can do this by running the following: |
443 |
|
444 |
$ wget https://code.grnet.gr/attachments/download/351/ganeti-instance-image_0.5.1-1-snf1_all.deb |
445 |
# dpkg -i ganeti-instance-image_0.5.1-1-snf1_all.deb |
446 |
|
447 |
Make any additional configuration changes in |
448 |
/etc/default/ganeti-instance-image. |
449 |
It is recommended to change the default IMAGE_DIR from |
450 |
/var/cache/ganeti-instance-image to something like /srv/example_images_repo |
451 |
|
452 |
Be sure to have all the package's needed dependencies installed |
453 |
on your system. |
454 |
|
455 |
2.From source |
456 |
|
457 |
Download and install gnt-instance-image from its own repository, |
458 |
at https://code.grnet.gr/git/gnt-instance-image. |
459 |
[FIXME: Must be updated to include information on installation of snf-image] |
460 |
|
461 |
Make sure to enable progress monitoring, using the --with-progress-monitor |
462 |
argument to configure. This requires the snf-progress-monitor tool, |
463 |
provided in snf-ganeti-tools/ and also as part of the snf-ganeti-tools |
464 |
Debian package. |
465 |
|
466 |
After installing gnt-instance-image do the following: |
467 |
1. $ cd /path-to-repo |
468 |
# cp ./defaults /etc/default/ganeti-instance-image |
469 |
2. In /etc/ganeti/instance-image/hooks, make sure the hooks you want to |
470 |
run during the instance creation process have execute permission. |
471 |
For linux you will need at lease `grub' and `root_passwd' to make the |
472 |
instance usable: |
473 |
chmod +x /etc/ganeti/instance-image/hooks/linux/{grub,root_passwd} |
474 |
For security reasons make sure `ssh' hook is also enabled. |
475 |
For windows you will need `mbr' and `admin_passwd': |
476 |
chmod +x /etc/ganeti/instance-image/hooks/windows/{mbr,admin_passwd} |
477 |
For both architectures it is also highly recommended to enable the |
478 |
`hostname' hook too: |
479 |
chmod +x /et/ganeti/instance-image/hooks/{linux,windows}/hostname |
480 |
|
481 |
Your custom Images should be stored in a dump format under |
482 |
/var/cache/ganeti-instance-image (default) or a different directory of your |
483 |
choice, accordingly set in /etc/default/ganeti-instance-image. The latter |
484 |
is recommended. Their filenames should have the following format: |
485 |
{backend_id}-x86_64-root.dump |
486 |
e.g., debian-6.0.1a-x86_64-root.dump (backend_id = "debian-6.0.1a") |
487 |
|
488 |
|
489 |
14. Setup Synnefo-specific networking on the Ganeti backend: |
490 |
This part is deployment-specific and must be customized based on the |
491 |
specific needs of the system administrators. |
492 |
|
493 |
A reference installation will use a Synnefo-specific KVM ifup script, |
494 |
NFDHCPD and pre-provisioned Linux bridges to support public and private |
495 |
network functionality. For this: |
496 |
|
497 |
Grab NFDHCPD from its own repository (https://code.grnet.gr/git/nfdhcpd), |
498 |
install it, modify /etc/nfdhcpd/nfdhcpd.conf to reflect your network |
499 |
configuration. |
500 |
|
501 |
Install a custom KVM ifup script for use by Ganeti, as |
502 |
/etc/ganeti/kvm-vif-bridge, on GANETI-NODEs. A sample implementation is |
503 |
provided under /contrib/ganeti-hooks. Set NFDHCPD_STATE_DIR to point |
504 |
to NFDHCPD's state directory, usually /var/lib/nfdhcpd. |
505 |
|
506 |
|
507 |
15. See section "Logging" in README.admin, and edit logic/logging.conf |
508 |
according to your OS and individual deployment characteristics. |
509 |
|
510 |
|
511 |
16. Optionally, read the okeanos_site/README file to setup ~okeanos introductory |
512 |
site (intro, video/info pages). Please see okeanos_site/90-okeanos.sample |
513 |
for a sample configuration file which overrides site-specific variables, |
514 |
to be placed under settings.d/, after customization. |
515 |
|
516 |
|
517 |
17. (Hopefully) Done |
518 |
|
519 |
|