Statistics
| Branch: | Tag: | Revision:

root / doc / source / install.rst @ 7aa06e71

History | View | Annotate | Download (16.1 kB)

1
************
2
Installation
3
************
4

    
5
.. toctree::
6
   :maxdepth: 2
7

    
8
Ubuntu 12.04.3 (64) - Django 1.3.x
9
==================================
10

    
11
This guide assumes that installation is carried out in /srv/flowspy directory. If other directory is to be used, please change the corresponding configuration files. It is also assumed that the root user will perform every action.
12

    
13
Required system packages
14
------------------------
15

    
16
Update and install the required packages::
17

    
18
	apt-get update
19
	apt-get upgrade
20
	apt-get install mysql-server apache2 memcached libapache2-mod-proxy-html gunicorn beanstalkd python-django python-django-extensions python-django-south python-django-tinymce tinymce python-mysqldb python-yaml python-memcache python-django-registration python-ipaddr python-lxml mysql-client git python-django-celery python-paramiko python-gevent vim
21

    
22
.. note::
23
 Set username and password for mysql if used
24

    
25
.. note::
26
	If you wish to deploy an outgoing mail server, now it is time to do it. Otherwise you could set FoD to send out mails via a third party account
27

    
28
Create a database
29
-----------------
30
If you are using mysql, you should create a database::
31

    
32
	mysql -u root -p -e 'create database fod'
33

    
34

    
35
Required application packages
36
-----------------------------
37
Get the required packages and install them
38

    
39
- ncclient: NETCONF python client::
40

    
41
	cd ~
42
	git clone https://github.com/leopoul/ncclient.git
43
	cd ncclient
44
	python setup.py install
45

    
46
- nxpy: Python Objects from/to XML proxy::
47

    
48
	cd ~
49
	git clone https://code.grnet.gr/git/nxpy
50
	cd nxpy
51
	python setup.py install
52

    
53
- flowspy: core application. Installation is done at /srv/flowspy::
54

    
55
	cd /srv
56
	git clone https://code.grnet.gr/git/flowspy
57
	cd flowspy
58

    
59
Application configuration
60
=========================
61

    
62
Copy settings.py.dist to settings.py::
63

    
64
	cp settings.py.dist settings.py
65

    
66
Edit settings.py file and set the following according to your configuration::
67

    
68
	ADMINS: set your admin name and email (assuming that your server can send notifications)
69
	DATABASES (to point to your local database). You could use views instead of tables for models: peer, peercontacts, peernetworks. For this to work we suggest MySQL with MyISAM db engine
70
	SECRET_KEY : Make this unique, and don't share it with anybody
71
	STATIC_URL (static media directory) . If you have followed the above this should be: /srv/flowspy/static
72
	TEMPLATE_DIRS : If you have followed the above this should be: /srv/flowspy/templates
73
	CACHE_BACKEND:  If you have followed the above this should be: memcached://127.0.0.1:11211/?timeout=3600
74
	Alternatively you could go for redis with the corresponding Django client lib.
75
	NETCONF_DEVICE (tested with Juniper EX4200 but any BGP enabled Juniper should work). This is the flowspec capable device
76
	NETCONF_USER (enable ssh and netconf on device)
77
	NETCONF_PASS
78
	If beanstalk is selected the following should be left intact.
79
	BROKER_HOST (beanstalk host)
80
	BROKER_PORT (beanstalk port)
81
	SERVER_EMAIL
82
	EMAIL_SUBJECT_PREFIX
83
	If beanstalk is selected the following should be left intact.
84
	BROKER_URL (beanstalk url)
85
	SHIB_AUTH_ENTITLEMENT (if you go for Shibboleth authentication)
86
	NOTIFY_ADMIN_MAILS (bcc mail addresses)
87
	PROTECTED_SUBNETS (subnets for which source or destination address will prevent rule creation and notify the NOTIFY_ADMIN_MAILS)
88
	The whois client is meant to be used in case you have inserted peers with their ASes in the peers table and wish to get network info for each one in an automated manner.
89
	PRIMARY_WHOIS
90
	ALTERNATE_WHOIS
91
	If you wish to deploy FoD with Shibboleth change the following attributes according to your setup:
92
	SHIB_AUTH_ENTITLEMENT = 'urn:mace'
93
	SHIB_ADMIN_DOMAIN = 'example.com'
94
	SHIB_LOGOUT_URL = 'https://example.com/Shibboleth.sso/Logout'
95
	SHIB_USERNAME = ['HTTP_EPPN']
96
	SHIB_MAIL = ['mail', 'HTTP_MAIL', 'HTTP_SHIB_INETORGPERSON_MAIL']
97
	SHIB_FIRSTNAME = ['HTTP_SHIB_INETORGPERSON_GIVENNAME']
98
	SHIB_LASTNAME = ['HTTP_SHIB_PERSON_SURNAME']
99
	SHIB_ENTITLEMENT = ['HTTP_SHIB_EP_ENTITLEMENT']
100

    
101
If you have not installed an outgoing mail server you can always use your own account (either corporate or gmail, hotmail ,etc) by adding the following lines in settings.py::
102

    
103
	EMAIL_USE_TLS = True #(or False)
104
	EMAIL_HOST = 'smtp.example.com'
105
	EMAIL_HOST_USER = 'username'
106
	EMAIL_HOST_PASSWORD = 'yourpassword'
107
	EMAIL_PORT = 587 #(outgoing)
108

    
109

    
110
.. note::
111
	Soon we will release a version with django-registration as a means to add users and Shibboleth as an alternative
112

    
113
Let's move on with some copies and dir creations::
114

    
115
	cp urls.py.dist urls.py
116
	mkdir log
117
	chown -R root:www-data log/
118
	chmod -R g+w log
119

    
120
System configuration
121
====================
122
Apache operates as a gunicorn Proxy with WSGI and Shibboleth modules enabled.
123
Depending on the setup the apache configuration may vary::
124

    
125
	a2enmod rewrite
126
	a2enmod proxy
127
	a2enmod ssl
128
	a2enmod proxy_http
129

    
130
If shibboleth is to be used::
131

    
132
	apt-get install libapache2-mod-shib2
133
	a2enmod shib2
134

    
135
Now it is time to configure beanstalk, gunicorn, celery and apache.
136

    
137
beanstalkd
138
----------
139

    
140
Enable beanstalk by editting /etc/default/beanstalkd::
141

    
142
	vim /etc/default/beanstalkd
143

    
144
Uncomment the line **START=yes** to enable beanstalk
145

    
146
Start beanstalkd::
147

    
148
	service beanstalkd start
149

    
150
gunicorn.d
151
----------
152

    
153
create and edit /etc/gunicorn.d/fod::
154

    
155
	vim /etc/gunicorn.d/fod
156

    
157
FoD is served via gunicorn and is then proxied by Apache. If the above directory conventions have been followed so far, then your configuration should be::
158

    
159
	CONFIG = {
160
	    'mode': 'django',
161
	    'working_dir': '/srv/flowspy',
162
	    'args': (
163
	        '--bind=127.0.0.1:8081',
164
	        '--workers=1',
165
	        '--timeout=360',
166
	        '--worker-class=egg:gunicorn#gevent',
167
	        '--log-level=debug',
168
	        'settings.py',
169
	    ),
170
	}
171

    
172
celery.d
173
--------
174

    
175
Celery is used over beanstalkd to apply firewall rules in a serial manner so that locks are avoided on the flowspec capable device. In our setup celery runs via django. That is why the python-django-celery package was installed.
176

    
177
Create the celeryd daemon at /etc/init.d/celeryd::
178

    
179
	vim /etc/init.d/celeryd
180

    
181
The configuration should be::
182

    
183
	#!/bin/sh -e
184
	# ============================================
185
	#  celeryd - Starts the Celery worker daemon.
186
	# ============================================
187
	#
188
	# :Usage: /etc/init.d/celeryd {start|stop|force-reload|restart|try-restart|status}
189
	# :Configuration file: /etc/default/celeryd
190
	#
191
	# See http://docs.celeryq.org/en/latest/cookbook/daemonizing.html#init-script-celeryd
192

    
193

    
194
	### BEGIN INIT INFO
195
	# Provides:          celeryd
196
	# Required-Start:    $network $local_fs $remote_fs
197
	# Required-Stop:     $network $local_fs $remote_fs
198
	# Default-Start:     2 3 4 5
199
	# Default-Stop:      0 1 6
200
	# Short-Description: celery task worker daemon
201
	### END INIT INFO
202

    
203
	#set -e
204

    
205
	DEFAULT_PID_FILE="/var/run/celeryd@%n.pid"
206
	DEFAULT_LOG_FILE="/var/log/celeryd@%n.log"
207
	DEFAULT_LOG_LEVEL="INFO"
208
	DEFAULT_NODES="celery"
209
	DEFAULT_CELERYD="-m celery.bin.celeryd_detach"
210

    
211
	# /etc/init.d/celeryd: start and stop the celery task worker daemon.
212

    
213
	CELERY_DEFAULTS=${CELERY_DEFAULTS:-"/etc/default/celeryd"}
214

    
215
	test -f "$CELERY_DEFAULTS" && . "$CELERY_DEFAULTS"
216
	if [ -f "/etc/default/celeryd" ]; then
217
	    . /etc/default/celeryd
218
	fi
219

    
220
	CELERYD_PID_FILE=${CELERYD_PID_FILE:-${CELERYD_PIDFILE:-$DEFAULT_PID_FILE}}
221
	CELERYD_LOG_FILE=${CELERYD_LOG_FILE:-${CELERYD_LOGFILE:-$DEFAULT_LOG_FILE}}
222
	CELERYD_LOG_LEVEL=${CELERYD_LOG_LEVEL:-${CELERYD_LOGLEVEL:-$DEFAULT_LOG_LEVEL}}
223
	CELERYD_MULTI=${CELERYD_MULTI:-"celeryd-multi"}
224
	CELERYD=${CELERYD:-$DEFAULT_CELERYD}
225
	CELERYCTL=${CELERYCTL:="celeryctl"}
226
	CELERYD_NODES=${CELERYD_NODES:-$DEFAULT_NODES}
227

    
228
	export CELERY_LOADER
229

    
230
	if [ -n "$2" ]; then
231
	    CELERYD_OPTS="$CELERYD_OPTS $2"
232
	fi
233

    
234
	CELERYD_LOG_DIR=`dirname $CELERYD_LOG_FILE`
235
	CELERYD_PID_DIR=`dirname $CELERYD_PID_FILE`
236
	if [ ! -d "$CELERYD_LOG_DIR" ]; then
237
	    mkdir -p $CELERYD_LOG_DIR
238
	fi
239
	if [ ! -d "$CELERYD_PID_DIR" ]; then
240
	    mkdir -p $CELERYD_PID_DIR
241
	fi
242

    
243
	# Extra start-stop-daemon options, like user/group.
244
	if [ -n "$CELERYD_USER" ]; then
245
	    DAEMON_OPTS="$DAEMON_OPTS --uid=$CELERYD_USER"
246
	    chown "$CELERYD_USER" $CELERYD_LOG_DIR $CELERYD_PID_DIR
247
	fi
248
	if [ -n "$CELERYD_GROUP" ]; then
249
	    DAEMON_OPTS="$DAEMON_OPTS --gid=$CELERYD_GROUP"
250
	    chgrp "$CELERYD_GROUP" $CELERYD_LOG_DIR $CELERYD_PID_DIR
251
	fi
252

    
253
	if [ -n "$CELERYD_CHDIR" ]; then
254
	    DAEMON_OPTS="$DAEMON_OPTS --workdir=\"$CELERYD_CHDIR\""
255
	fi
256

    
257

    
258
	check_dev_null() {
259
	    if [ ! -c /dev/null ]; then
260
		echo "/dev/null is not a character device!"
261
		exit 1
262
	    fi
263
	}
264

    
265

    
266
	export PATH="${PATH:+$PATH:}/usr/sbin:/sbin"
267

    
268

    
269
	stop_workers () {
270
	    $CELERYD_MULTI stop $CELERYD_NODES --pidfile="$CELERYD_PID_FILE"
271
	}
272

    
273

    
274
	start_workers () {
275
	    $CELERYD_MULTI start $CELERYD_NODES $DAEMON_OPTS        \
276
		                 --pidfile="$CELERYD_PID_FILE"      \
277
		                 --logfile="$CELERYD_LOG_FILE"      \
278
		                 --loglevel="$CELERYD_LOG_LEVEL"    \
279
		                 --cmd="$CELERYD"                   \
280
		                 $CELERYD_OPTS
281
	}
282

    
283

    
284
	restart_workers () {
285
	    $CELERYD_MULTI restart $CELERYD_NODES $DAEMON_OPTS      \
286
		                   --pidfile="$CELERYD_PID_FILE"    \
287
		                   --logfile="$CELERYD_LOG_FILE"    \
288
		                   --loglevel="$CELERYD_LOG_LEVEL"  \
289
		                   --cmd="$CELERYD"                 \
290
		                   $CELERYD_OPTS
291
	}
292

    
293

    
294

    
295
	case "$1" in
296
	    start)
297
		check_dev_null
298
		start_workers
299
	    ;;
300

    
301
	    stop)
302
		check_dev_null
303
		stop_workers
304
	    ;;
305

    
306
	    reload|force-reload)
307
		echo "Use restart"
308
	    ;;
309

    
310
	    status)
311
		$CELERYCTL status $CELERYCTL_OPTS
312
	    ;;
313

    
314
	    restart)
315
		check_dev_null
316
		restart_workers
317
	    ;;
318

    
319
	    try-restart)
320
		check_dev_null
321
		restart_workers
322
	    ;;
323

    
324
	    *)
325
		echo "Usage: /etc/init.d/celeryd {start|stop|restart|try-restart|kill}"
326
		exit 1
327
	    ;;
328
	esac
329

    
330
	exit 0
331

    
332
and make it executable::
333

    
334
	chmod +x /etc/init.d/celeryd
335

    
336
celeryd requires a /etc/default/celeryd file to be in place.
337
Thus we are going to create this file (/etc/default/celeryd)::
338

    
339
	vim /etc/default/celeryd
340

    
341
Again if the directory conventions have been followed the file should be::
342

    
343
	# Name of nodes to start, here we have a single node
344
	CELERYD_NODES="w1"
345
	# or we could have three nodes:
346
	#CELERYD_NODES="w1 w2 w3"
347

    
348
	# Where to chdir at start.
349
	CELERYD_CHDIR="/srv/flowspy/"
350
	# How to call "manage.py celeryd_multi"
351
	CELERYD_MULTI="$CELERYD_CHDIR/manage.py celeryd_multi"
352

    
353
	# How to call "manage.py celeryctl"
354
	CELERYCTL="$CELERYD_CHDIR/manage.py celeryctl"
355

    
356
	# Extra arguments to celeryd
357
	#CELERYD_OPTS="--time-limit=300 --concurrency=8"
358
	CELERYD_OPTS="-E -B"
359
	# Name of the celery config module.
360
	CELERY_CONFIG_MODULE="celeryconfig"
361

    
362
	# %n will be replaced with the nodename.
363
	CELERYD_LOG_FILE="$CELERYD_CHDIR/celery_var/log/celery/%n.log"
364
	CELERYD_PID_FILE="$CELERYD_CHDIR/celery_var/run/celery/%n.pid"
365

    
366
	# Workers should run as an unprivileged user.
367
	CELERYD_USER="root"
368
	CELERYD_GROUP="root"
369

    
370
	# Name of the projects settings module.
371
	export DJANGO_SETTINGS_MODULE="settings"
372

    
373
Apache
374
------
375
Apache proxies gunicorn. Things are more flexible here as you may follow your own configuration and conventions. Create and edit /etc/apache2/sites-available/fod. You should set <server_name> and <admin_mail> along with your certificates. If under testing environment, you can use the provided snakeoil certs. If you do not intent to use Shibboleth delete or comment the corresponding configuration parts inside **Shibboleth configuration** ::
376

    
377
	vim /etc/apache2/sites-available/fod
378

    
379
Again if the directory conventions have been followed the file should be::
380

    
381
	<VirtualHost *:80>
382
		ServerAdmin webmaster@localhost
383
		ServerName	<server_name>
384
		DocumentRoot /var/www
385
		<Directory />
386
			Options FollowSymLinks
387
			AllowOverride None
388
		</Directory>
389
		<Directory /var/www/>
390
			Options Indexes FollowSymLinks MultiViews
391
			AllowOverride None
392
			Order allow,deny
393
			allow from all
394
		</Directory>
395

    
396
		ScriptAlias /cgi-bin/ /usr/lib/cgi-bin/
397
		<Directory "/usr/lib/cgi-bin">
398
			AllowOverride None
399
			Options +ExecCGI -MultiViews +SymLinksIfOwnerMatch
400
			Order allow,deny
401
			Allow from all
402
		</Directory>
403

    
404
		ErrorLog ${APACHE_LOG_DIR}/error.log
405

    
406
		# Possible values include: debug, info, notice, warn, error, crit,
407
		# alert, emerg.
408
		LogLevel warn
409

    
410
		CustomLog ${APACHE_LOG_DIR}/access.log combined
411

    
412
	    Alias /doc/ "/usr/share/doc/"
413
	    <Directory "/usr/share/doc/">
414
	        Options Indexes MultiViews FollowSymLinks
415
	        AllowOverride None
416
	        Order deny,allow
417
	        Deny from all
418
	        Allow from 127.0.0.0/255.0.0.0 ::1/128
419
	    </Directory>
420

    
421
		RewriteEngine On
422
		RewriteCond %{HTTPS} off
423
		RewriteRule ^/(.*) https://<server_name>/$1 [L,R]
424

    
425
	</VirtualHost>
426
	<VirtualHost *:443>
427
		ServerName	 <server_name>
428
		ServerAdmin		<admin_mail>
429
		ServerSignature		On
430

    
431
		SSLEngine on
432
		SSLCertificateFile	/etc/ssl/certs/example.com.crt
433
		SSLCertificateChainFile	/etc/ssl/certs/example.com.crt
434
		SSLCertificateKeyFile	/etc/ssl/private/example.com.key
435

    
436
		AddDefaultCharset	UTF-8
437
		IndexOptions		+Charset=UTF-8
438

    
439
		# Shibboleth configuration
440
		ShibConfig		/etc/shibboleth/shibboleth2.xml
441
		Alias			/shibboleth-sp	/usr/share/shibboleth
442

    
443
		<Location /fod/login>
444
	        AuthType shibboleth
445
	        ShibRequireSession On
446
	        ShibUseHeaders On
447
	        require valid-user
448
	    </Location>
449

    
450
		# Shibboleth debugging CGI script
451
		ScriptAlias /shibboleth/test /usr/lib/cgi-bin/shibtest.cgi
452
		<Location /shibboleth/test>
453
			AuthType shibboleth
454
			ShibRequireSession On
455
			ShibUseHeaders On
456
			require valid-user
457
		</Location>
458

    
459
		<Location /Shibboleth.sso>
460
			SetHandler shib
461
		</Location>
462

    
463
		# End of Shibboleth configuration
464

    
465
		<Location /admin/media/>
466
			SetHandler      None
467
		</Location>
468

    
469
		Alias /admin/media	/usr/share/pyshared/django/contrib/admin/media
470
		Alias /media  /usr/share/pyshared/django/contrib/admin/media
471
			DocumentRoot /var/www
472
	        <Directory /var/www/>
473
	                Options Indexes FollowSymLinks MultiViews
474
	                AllowOverride None
475
	                Order allow,deny
476
	                allow from all
477
	        </Directory>
478

    
479

    
480

    
481
		<Proxy *>
482
	  	Order allow,deny
483
		Allow from all
484
		</Proxy>
485

    
486
		SSLProxyEngine        off
487
		ProxyErrorOverride    off
488
	    ProxyTimeout    28800
489
		ProxyPass        /fod http://localhost:8081/fod retry=0
490
		ProxyPassReverse /fod http://localhost:8081/fod
491

    
492
		LogLevel warn
493
		ErrorLog /var/log/apache2/ssl-error.log
494
		CustomLog /var/log/apache2/ssl-access.log combined
495

    
496

    
497

    
498

    
499
		Alias /fodstatic	/srv/flowspy/static
500

    
501
	</VirtualHost>
502

    
503
Now, enable your site. You might want to disable the default site if fod is the only site you host on your server::
504

    
505
	a2dissite default
506
	a2ensite fod
507

    
508
You are not far away from deploying FoD. When asked for a super user, create one::
509

    
510
	cd /srv/flowspy
511
	python manage.py syncdb
512
	python manage.py migrate
513

    
514
Restart, gunicorn and apache::
515

    
516
	service gunicorn restart && service apache2 restart
517

    
518
Testing the platform
519
====================
520
Log in to the admin interface via https://<your ip>/fod/admin. Go to Peer ranges and add a new range (part of/or a complete subnet), eg. 83.212.0.0/19
521
Go to Peers and add a new peer, eg. id: 1, name: Test, AS: 16503, tag: TEST and move the network you have crteated from Avalable to Chosen. From the admin front, go to User, and edit your user. From the bottom of the page, select the TEST peer and save.
522
Last but not least, modify as required the existing (example.com) Site instance (admin home->Sites). You are done. As you are logged-in via the admin, there is no need for Shibboleth. Go to https://<your ip>/fod/ and create a new rule. Your rule should be applied on the flowspec capable device after aprox. 10 seconds.
523

    
524
Branding
525
========
526
Via the admin interface you can modify flatpages to suit your needs
527

    
528
Logos
529
-----
530
Inside the static folder you will find two empty png files: fod_logo.xcf (Gimp file) and shib_login.dist.png. Edit those two with your favourite image processing software and save them as fod_logo.png (under static/img/) and shib_login.png (under static/). Image sizes are optimized to operate without any other code changes. In case you want to incorporate images of different sizes you have to fine tune css and/or html as well.
531

    
532
Footer
533
------
534
Under the templates folder (templates), you can alter the footer.html file to include your own footer messages, badges, etc.
535

    
536
Welcome Page
537
------------
538
Under the templates folder (templates), you can alter the welcome page - welcome.html with your own images, carousel, videos, etc.