Statistics
| Branch: | Tag: | Revision:

root / doc / source / install.rst @ ca345a18

History | View | Annotate | Download (19 kB)

1
************
2
Installation
3
************
4

    
5
.. toctree::
6
    :maxdepth: 2
7

    
8
Debian Wheezy (x64) - Django 1.4.x
9
==================================
10
This guide assumes that installation is carried out in /srv/flowspy directory. If other directory is to be used, please change the corresponding configuration files. It is also assumed that the root user will perform every action.
11

    
12
Upgrading from v<1.0.x
13
----------------------
14
If upgrading from flowspy version <1.0.x pay attention to settings.py changes. Also, do not forget to run::
15
    
16
    ./manage.py migrate
17
    
18
to catch-up with latest database changes.
19

    
20
Required system packages
21
------------------------
22
Update and install the required packages::
23

    
24
    apt-get update
25
    apt-get upgrade
26
    apt-get install mysql-server apache2 memcached libapache2-mod-proxy-html gunicorn beanstalkd python-django python-django-south python-django-tinymce tinymce python-mysqldb python-yaml python-memcache python-django-registration python-ipaddr python-lxml mysql-client git python-django-celery python-paramiko python-gevent vim
27

    
28
.. note::
29
    Set username and password for mysql if used
30

    
31
.. note::
32
    If you wish to deploy an outgoing mail server, now it is time to do it. Otherwise you could set FoD to send out mails via a third party account
33

    
34
Create a database
35
-----------------
36
If you are using mysql, you should create a database::
37

    
38
    mysql -u root -p -e 'create database fod'
39

    
40
Required application packages
41
-----------------------------
42
Get the required packages and their dependencies and install them::
43

    
44
    apt-get install libxml2-dev libxslt-dev gcc python-dev
45

    
46
- ncclient: NETCONF python client::
47

    
48
    cd ~
49
    git clone https://github.com/leopoul/ncclient.git
50
    cd ncclient
51
    python setup.py install
52

    
53
- nxpy: Python Objects from/to XML proxy::
54

    
55
    cd ~
56
    git clone https://code.grnet.gr/git/nxpy
57
    cd nxpy
58
    python setup.py install
59

    
60
- flowspy: core application. Installation is done at /srv/flowspy::
61

    
62
    cd /srv
63
    git clone https://code.grnet.gr/git/flowspy
64
    cd flowspy
65

    
66
Application configuration
67
=========================
68
Copy settings.py.dist to settings.py::
69
    
70
    cd flowspy
71
    cp settings.py.dist settings.py
72

    
73
Edit settings.py file and set the following according to your configuration::
74

    
75
    ADMINS: set your admin name and email (assuming that your server can send notifications)
76
    DATABASES (to point to your local database). You could use views instead of tables for models: peer, peercontacts, peernetworks. For this to work we suggest MySQL with MyISAM db engine
77
    SECRET_KEY : Make this unique, and don't share it with anybody
78
    STATIC_ROOT: /srv/flowspy/static (or your installation directory)
79
    STATIC_URL (static media directory) . If you have followed the above this should be: /srv/flowspy/static
80
    TEMPLATE_DIRS : If you have followed the above this should be: /srv/flowspy/templates
81
    CACHE_BACKEND:  Enable Memcached for production or leave to DummyCache for development environments
82
    Alternatively you could go for redis with the corresponding Django client lib.
83
    NETCONF_DEVICE (tested with Juniper EX4200 but any BGP enabled Juniper should work). This is the flowspec capable device
84
    NETCONF_USER (enable ssh and netconf on device)
85
    NETCONF_PASS
86
    If beanstalk is selected the following should be left intact.
87
    BROKER_HOST (beanstalk host)
88
    BROKER_PORT (beanstalk port)
89
    SERVER_EMAIL
90
    EMAIL_SUBJECT_PREFIX
91
    If beanstalk is selected the following should be left intact.
92
    BROKER_URL (beanstalk url)
93
    SHIB_AUTH_ENTITLEMENT (if you go for Shibboleth authentication)
94
    NOTIFY_ADMIN_MAILS (bcc mail addresses)
95
    PROTECTED_SUBNETS (subnets for which source or destination address will prevent rule creation and notify the NOTIFY_ADMIN_MAILS)
96
    The whois client is meant to be used in case you have inserted peers with their ASes in the peers table and wish to get network info for each one in an automated manner.
97
    PRIMARY_WHOIS
98
    ALTERNATE_WHOIS
99
    If you wish to deploy FoD with Shibboleth change the following attributes according to your setup:
100
    SHIB_AUTH_ENTITLEMENT = 'urn:mace'
101
    SHIB_ADMIN_DOMAIN = 'example.com'
102
    SHIB_LOGOUT_URL = 'https://example.com/Shibboleth.sso/Logout'
103
    SHIB_USERNAME = ['HTTP_EPPN']
104
    SHIB_MAIL = ['mail', 'HTTP_MAIL', 'HTTP_SHIB_INETORGPERSON_MAIL']
105
    SHIB_FIRSTNAME = ['HTTP_SHIB_INETORGPERSON_GIVENNAME']
106
    SHIB_LASTNAME = ['HTTP_SHIB_PERSON_SURNAME']
107
    SHIB_ENTITLEMENT = ['HTTP_SHIB_EP_ENTITLEMENT']
108

    
109
If you have not installed an outgoing mail server you can always use your own account (either corporate or gmail, hotmail ,etc) by adding the following lines in settings.py::
110

    
111
    EMAIL_USE_TLS = True #(or False)
112
    EMAIL_HOST = 'smtp.example.com'
113
    EMAIL_HOST_USER = 'username'
114
    EMAIL_HOST_PASSWORD = 'yourpassword'
115
    EMAIL_PORT = 587 #(outgoing)
116

    
117
It is strongly advised that you do not change the following to False values unless, you want to integrate FoD with you CRM or members database. This implies that you are able/have the rights to create database views between the two databases::
118

    
119
    PEER_MANAGED_TABLE = True
120
    PEER_RANGE_MANAGED_TABLE = True
121
    PEER_TECHC_MANAGED_TABLE = True   
122

    
123
By doing that the corresponding tables as defined in peers/models will not be created. As noted above, you have to create the views that the tables will rely on. 
124

    
125
.. note::
126
    Soon we will release a version with django-registration as a means to add users and Shibboleth will become an alternative
127

    
128
Let's move on with some copies and dir creations::
129

    
130
    mkdir /var/log/fod
131
    chown www-data.www-data /var/log/fod
132
    cp urls.py.dist urls.py
133
    cd ..
134

    
135
System configuration
136
====================
137
Apache operates as a gunicorn Proxy with WSGI and Shibboleth modules enabled.
138
Depending on the setup the apache configuration may vary::
139

    
140
    a2enmod rewrite
141
    a2enmod proxy
142
    a2enmod ssl
143
    a2enmod proxy_http
144

    
145
If shibboleth is to be used::
146

    
147
    apt-get install libapache2-mod-shib2
148
    a2enmod shib2
149

    
150
Now it is time to configure beanstalk, gunicorn, celery and apache.
151

    
152
beanstalkd
153
----------
154
Enable beanstalk by editting /etc/default/beanstalkd::
155

    
156
    vim /etc/default/beanstalkd
157

    
158
Uncomment the line **START=yes** to enable beanstalk
159

    
160
Start beanstalkd::
161

    
162
    service beanstalkd start
163

    
164
gunicorn.d
165
----------
166
Create and edit /etc/gunicorn.d/fod::
167

    
168
    vim /etc/gunicorn.d/fod
169

    
170
FoD is served via gunicorn and is then proxied by Apache. If the above directory conventions have been followed so far, then your configuration should be::
171

    
172
    CONFIG = {
173
          'mode': 'django',
174
          'working_dir': '/srv/flowspy',
175
          'args': (
176
               '--bind=127.0.0.1:8081',
177
               '--workers=1',
178
               '--worker-class=egg:gunicorn#gevent',
179
               '--timeout=30',
180
               '--debug',
181
               '--log-level=debug',
182
               '--log-file=/var/log/gunicorn/fod.log',
183
          ),
184
    }
185

    
186

    
187
celeryd
188
-------
189
Celery is used over beanstalkd to apply firewall rules in a serial manner so that locks are avoided on the flowspec capable device. In our setup celery runs via django. That is why the python-django-celery package was installed.
190

    
191
Create the celeryd daemon at /etc/init.d/celeryd **if it does not already exist**::
192

    
193
    vim /etc/init.d/celeryd
194

    
195
The configuration should be::
196

    
197
    #!/bin/sh -e
198
    # ============================================
199
    #  celeryd - Starts the Celery worker daemon.
200
    # ============================================
201
    #
202
    # :Usage: /etc/init.d/celeryd {start|stop|force-reload|restart|try-restart|status}
203
    # :Configuration file: /etc/default/celeryd
204
    #
205
    # See http://docs.celeryq.org/en/latest/cookbook/daemonizing.html#init-script-celeryd
206
    
207
    
208
    ### BEGIN INIT INFO
209
    # Provides:              celeryd
210
    # Required-Start:     $network $local_fs $remote_fs
211
    # Required-Stop:       $network $local_fs $remote_fs
212
    # Default-Start:       2 3 4 5
213
    # Default-Stop:        0 1 6
214
    # Short-Description: celery task worker daemon
215
    # Description:          Starts the Celery worker daemon for a single project.
216
    ### END INIT INFO
217
    
218
    #set -e
219
    
220
    DEFAULT_PID_FILE="/var/run/celery/%n.pid"
221
    DEFAULT_LOG_FILE="/var/log/celery/%n.log"
222
    DEFAULT_LOG_LEVEL="INFO"
223
    DEFAULT_NODES="celery"
224
    DEFAULT_CELERYD="-m celery.bin.celeryd_detach"
225
    ENABLED="false"
226
    
227
    [ -r "$CELERY_DEFAULTS" ] && . "$CELERY_DEFAULTS"
228
    
229
    [ -r /etc/default/celeryd ] && . /etc/default/celeryd
230
    
231
    if [ "$ENABLED" != "true" ]; then
232
          echo "celery daemon disabled - see /etc/default/celeryd."
233
          exit 0
234
    fi
235
    
236
    
237
    CELERYD_PID_FILE=${CELERYD_PID_FILE:-${CELERYD_PIDFILE:-$DEFAULT_PID_FILE}}
238
    CELERYD_LOG_FILE=${CELERYD_LOG_FILE:-${CELERYD_LOGFILE:-$DEFAULT_LOG_FILE}}
239
    CELERYD_LOG_LEVEL=${CELERYD_LOG_LEVEL:-${CELERYD_LOGLEVEL:-$DEFAULT_LOG_LEVEL}}
240
    CELERYD_MULTI=${CELERYD_MULTI:-"celeryd-multi"}
241
    CELERYD=${CELERYD:-$DEFAULT_CELERYD}
242
    CELERYCTL=${CELERYCTL:="celeryctl"}
243
    CELERYD_NODES=${CELERYD_NODES:-$DEFAULT_NODES}
244
    
245
    export CELERY_LOADER
246
    
247
    if [ -n "$2" ]; then
248
          CELERYD_OPTS="$CELERYD_OPTS $2"
249
    fi
250
    
251
    CELERYD_LOG_DIR=`dirname $CELERYD_LOG_FILE`
252
    CELERYD_PID_DIR=`dirname $CELERYD_PID_FILE`
253
    if [ ! -d "$CELERYD_LOG_DIR" ]; then
254
          mkdir -p $CELERYD_LOG_DIR
255
    fi
256
    if [ ! -d "$CELERYD_PID_DIR" ]; then
257
          mkdir -p $CELERYD_PID_DIR
258
    fi
259
    
260
    # Extra start-stop-daemon options, like user/group.
261
    if [ -n "$CELERYD_USER" ]; then
262
          DAEMON_OPTS="$DAEMON_OPTS --uid=$CELERYD_USER"
263
          chown "$CELERYD_USER" $CELERYD_LOG_DIR $CELERYD_PID_DIR
264
    fi
265
    if [ -n "$CELERYD_GROUP" ]; then
266
          DAEMON_OPTS="$DAEMON_OPTS --gid=$CELERYD_GROUP"
267
          chgrp "$CELERYD_GROUP" $CELERYD_LOG_DIR $CELERYD_PID_DIR
268
    fi
269
    
270
    if [ -n "$CELERYD_CHDIR" ]; then
271
          DAEMON_OPTS="$DAEMON_OPTS --workdir=\"$CELERYD_CHDIR\""
272
    fi
273
    
274
    
275
    check_dev_null() {
276
          if [ ! -c /dev/null ]; then
277
               echo "/dev/null is not a character device!"
278
               exit 1
279
          fi
280
    }
281
    
282
    
283
    export PATH="${PATH:+$PATH:}/usr/sbin:/sbin"
284
    
285
    
286
    stop_workers () {
287
          $CELERYD_MULTI stop $CELERYD_NODES --pidfile="$CELERYD_PID_FILE"
288
    }
289
    
290
    
291
    start_workers () {
292
          $CELERYD_MULTI start $CELERYD_NODES $DAEMON_OPTS           \
293
                                        --pidfile="$CELERYD_PID_FILE"        \
294
                                        --logfile="$CELERYD_LOG_FILE"        \
295
                                        --loglevel="$CELERYD_LOG_LEVEL"     \
296
                                        --cmd="$CELERYD"                           \
297
                                        $CELERYD_OPTS
298
    }
299
    
300
    
301
    restart_workers () {
302
          $CELERYD_MULTI restart $CELERYD_NODES $DAEMON_OPTS        \
303
                                           --pidfile="$CELERYD_PID_FILE"     \
304
                                           --logfile="$CELERYD_LOG_FILE"     \
305
                                           --loglevel="$CELERYD_LOG_LEVEL"  \
306
                                           --cmd="$CELERYD"                        \
307
                                           $CELERYD_OPTS
308
    }
309
    
310
    
311
    
312
    case "$1" in
313
          start)
314
               check_dev_null
315
               start_workers
316
          ;;
317
    
318
          stop)
319
               check_dev_null
320
               stop_workers
321
          ;;
322
    
323
          reload|force-reload)
324
               echo "Use restart"
325
          ;;
326
    
327
          status)
328
               $CELERYCTL status $CELERYCTL_OPTS
329
          ;;
330
    
331
          restart)
332
               check_dev_null
333
               restart_workers
334
          ;;
335
    
336
          try-restart)
337
               check_dev_null
338
               restart_workers
339
          ;;
340
    
341
          *)
342
               echo "Usage: /etc/init.d/celeryd {start|stop|restart|try-restart|kill}"
343
               exit 1
344
          ;;
345
    esac
346
    
347
    exit 0
348

    
349
celeryd configuration
350
---------------------
351
celeryd requires a /etc/default/celeryd file to be in place.
352
Thus we are going to create this file (/etc/default/celeryd)::
353

    
354
    vim /etc/default/celeryd
355

    
356
Again if the directory conventions have been followed the file is (pay attention to the CELERYD_USER, CELERYD_GROUP and change accordingly)  ::
357

    
358
    # Default: false
359
    ENABLED="true"
360
    
361
    # Name of nodes to start, here we have a single node
362
    CELERYD_NODES="w1"
363
    # or we could have three nodes:
364
    #CELERYD_NODES="w1 w2 w3"
365
    
366
    # Where to chdir at start.
367
    CELERYD_CHDIR="/srv/flowspy"
368
    # How to call "manage.py celeryd_multi"
369
    CELERYD_MULTI="python $CELERYD_CHDIR/manage.py celeryd_multi"
370
    
371
    # How to call "manage.py celeryctl"
372
    CELERYCTL="python $CELERYD_CHDIR/manage.py celeryctl"
373
    
374
    # Extra arguments to celeryd
375
    #CELERYD_OPTS="--time-limit=300 --concurrency=8"
376
    CELERYD_OPTS="-E -B --schedule=/var/run/celery/celerybeat-schedule --concurrency=1 --soft-time-limit=180 --time-limit=1800"
377
    # Name of the celery config module.
378
    CELERY_CONFIG_MODULE="celeryconfig"
379
    
380
    # %n will be replaced with the nodename.
381
    CELERYD_LOG_FILE="/var/log/celery/fod_%n.log"
382
    CELERYD_PID_FILE="/var/run/celery/%n.pid"
383
    
384
    # Workers should run as an unprivileged user.
385
    CELERYD_USER="celery"
386
    CELERYD_GROUP="celery"
387
    
388
    # Name of the projects settings module.
389
    export DJANGO_SETTINGS_MODULE="flowspy.settings"
390

    
391
Apache
392
------
393
Apache proxies gunicorn. Things are more flexible here as you may follow your own configuration and conventions. Create and edit /etc/apache2/sites-available/fod. You should set <server_name> and <admin_mail> along with your certificates. If under testing environment, you can use the provided snakeoil certs. If you do not intent to use Shibboleth delete or comment the corresponding configuration parts inside **Shibboleth configuration** ::
394

    
395
    vim /etc/apache2/sites-available/fod
396

    
397
Again if the directory conventions have been followed the file should be::
398

    
399
    <VirtualHost *:80>
400
        ServerAdmin webmaster@localhost
401
        ServerName  fod.example.com
402
        DocumentRoot /var/www
403
    
404
        ErrorLog ${APACHE_LOG_DIR}/fod_error.log
405
    
406
        # Possible values include: debug, info, notice, warn, error, crit,
407
        # alert, emerg.
408
        LogLevel debug
409
        
410
        CustomLog ${APACHE_LOG_DIR}/fod_access.log combined
411
    
412
        Alias /static       /srv/flowspy/static
413
          RewriteEngine On
414
          RewriteCond %{HTTPS} off
415
          RewriteRule ^/(.*) https://fod.example.com/$1 [L,R]
416
    </VirtualHost>
417
    
418
    <VirtualHost *:443>
419
        ServerName    fod.example.com
420
        ServerAdmin     webmaster@localhost
421
        ServerSignature        On
422
        
423
        SSLEngine on
424
        SSLCertificateFile    /etc/ssl/certs/fod.example.com.crt
425
        SSLCertificateChainFile /etc/ssl/certs/example-chain.pem
426
        SSLCertificateKeyFile    /etc/ssl/private/fod.example.com.key
427
    
428
        AddDefaultCharset UTF-8
429
        IndexOptions        +Charset=UTF-8
430
    
431
        ShibConfig       /etc/shibboleth/shibboleth2.xml
432
        Alias          /shibboleth-sp /usr/share/shibboleth
433
    
434
    
435
        <Location /login>
436
             AuthType shibboleth
437
             ShibRequireSession On
438
             ShibUseHeaders On
439
             ShibRequestSetting entityID https://idp.example.com/idp/shibboleth
440
             require valid-user
441
        </Location>
442
        
443
        # Shibboleth debugging CGI script
444
        ScriptAlias /shibboleth/test /usr/lib/cgi-bin/shibtest.cgi
445
        <Location /shibboleth/test>
446
             AuthType shibboleth
447
             ShibRequireSession On
448
             ShibUseHeaders On
449
             require valid-user
450
        </Location>
451
    
452
        <Location /Shibboleth.sso>
453
             SetHandler shib
454
        </Location>
455
    
456
        # Shibboleth SP configuration
457
    
458
        #SetEnv                       proxy-sendchunked
459
        
460
              <Proxy *>
461
               Order allow,deny
462
               Allow from all
463
               </Proxy>
464
    
465
               SSLProxyEngine           off
466
               ProxyErrorOverride     off
467
          ProxyTimeout     28800
468
             ProxyPass        /static !
469
             ProxyPass          /shibboleth !
470
             ProxyPass        /Shibboleth.sso !
471
             
472
               ProxyPass           / http://localhost:8081/ retry=0
473
               ProxyPassReverse / http://localhost:8081/
474
    
475
          Alias /static          /srv/flowspy/static
476
    
477
        LogLevel warn
478
        
479
        ErrorLog ${APACHE_LOG_DIR}/fod_error.log
480
          CustomLog ${APACHE_LOG_DIR}/fod_access.log combined
481
    
482
    </VirtualHost>
483

    
484
Now, enable your site. You might want to disable the default site if fod is the only site you host on your server::
485

    
486
    a2dissite default
487
    a2ensite fod
488

    
489
You are not far away from deploying FoD. When asked for a super user, create one::
490

    
491
    cd /srv/flowspy
492
    python manage.py syncdb
493
    python manage.py migrate flowspec
494
    python manage.py migrate djcelery
495
    python manage.py migrate accounts
496

    
497
If you have not changed the values of the PEER\_\*\_TABLE variables to False and thus you are going for a default installation (that is PEER\_\*\_TABLE variables are set to True) , then run::
498
    
499
    python manage.py migrate peers
500

    
501
If however you have set the PEER\_\*\_TABLE variables to False and by accident you have ran the command above, then you have to cleanup you database manually by dropping the peer\* tables plus the techc_email table. For MySQL the command is::
502
    
503
    DROP TABLE `peer`, `peer_networks`, `peer_range`, `peer_techc_emails`, techc_email;  
504

    
505
Restart, gunicorn and apache::
506

    
507
    service gunicorn restart && service apache2 restart
508

    
509

    
510
Propagate the flatpages
511
=======================
512
Inside the initial_data/fixtures_manual.xml file we have placed 4 flatpages (2 for Greek, 2 for English) with Information and Terms of Service about the service. 
513
To import the flatpages, run from root folder::
514

    
515
    python manage.py loaddata initial_data/fixtures_manual.xml
516

    
517

    
518

    
519
Testing the platform
520
====================
521
Log in to the admin interface via https:\/\/<hostname>\/admin. Go to Peer ranges and add a new range (part of/or a complete subnet), eg. 10.20.0.0/19
522
Go to Peers and add a new peer, eg. id: 1, name: Test, AS: 16503, tag: TEST and move the network you have created from Avalable to Chosen. From the admin front, go to User, and edit your user. From the bottom of the page, select the TEST peer and save.
523
Last but not least, modify as required the existing (example.com) Site instance (admin home->Sites). You are done. As you are logged-in via the admin, there is no need to go through Shibboleth at this time. Go to https:\/\/<hostname>\/ and create a new rule. Your rule should be applied on the flowspec capable device after aprox. 10 seconds. If no Shibboleth authentication is available, a https:\/\/<hostname>\/altlogin is provided.
524

    
525
Branding
526
========
527
Via the admin interface you can modify flatpages to suit your needs
528

    
529
Footer
530
------
531
Under the templates folder (templates), you can alter the footer.html file to include your own footer messages, badges, etc.
532

    
533
Welcome Page
534
------------
535
Under the templates folder (templates), you can alter the welcome page - welcome.html with your own images, carousel, videos, etc.