Statistics
| Branch: | Tag: | Revision:

root / doc / source / install.rst @ ac2fbadf

History | View | Annotate | Download (19.6 kB)

1
************
2
Installation
3
************
4

    
5
.. toctree::
6
    :maxdepth: 2
7

    
8
Debian Wheezy (x64) - Django 1.4.x
9
==================================
10
This guide assumes that installation is carried out in /srv/flowspy directory. If other directory is to be used, please change the corresponding configuration files. It is also assumed that the root user will perform every action.
11

    
12

    
13
Upgrading from v<1.1.x
14
----------------------
15

    
16
.. note::
17
    If PEER\_\*\_TABLE tables are set to FALSE in settings.py, you need to perform the south migrations per application:: 
18
    
19
        ./manage.py migrate longerusername
20
        ./manage.py migrate flowspec
21
        ./manage.py migrate accounts
22
        
23
        
24

    
25
If upgrading from flowspy version <1.1.x pay attention to settings.py changes. Also, do not forget to run if PEER\_\*\_TABLE tables are set to TRUE in settings.py::
26
    
27
    ./manage.py migrate 
28
    
29
to catch-up with latest database changes. 
30

    
31
Upgrading from v<1.0.x
32
----------------------
33
If upgrading from flowspy version <1.0.x pay attention to settings.py changes. Also, do not forget to run::
34
    
35
    ./manage.py migrate
36
    
37
to catch-up with latest database changes.
38

    
39
Required system packages
40
------------------------
41
Update and install the required packages::
42

    
43
    apt-get update
44
    apt-get upgrade
45
    apt-get install mysql-server apache2 memcached libapache2-mod-proxy-html gunicorn beanstalkd python-django python-django-south python-django-tinymce tinymce python-mysqldb python-yaml python-memcache python-django-registration python-ipaddr python-lxml mysql-client git python-django-celery python-paramiko python-gevent vim
46

    
47
.. note::
48
    Set username and password for mysql if used
49

    
50
.. note::
51
    If you wish to deploy an outgoing mail server, now it is time to do it. Otherwise you could set FoD to send out mails via a third party account
52

    
53
Create a database
54
-----------------
55
If you are using mysql, you should create a database::
56

    
57
    mysql -u root -p -e 'create database fod'
58

    
59
Required application packages
60
-----------------------------
61
Get the required packages and their dependencies and install them::
62

    
63
    apt-get install libxml2-dev libxslt-dev gcc python-dev
64

    
65
- ncclient: NETCONF python client::
66

    
67
    cd ~
68
    git clone https://github.com/leopoul/ncclient.git
69
    cd ncclient
70
    python setup.py install
71

    
72
- nxpy: Python Objects from/to XML proxy::
73

    
74
    cd ~
75
    git clone https://code.grnet.gr/git/nxpy
76
    cd nxpy
77
    python setup.py install
78

    
79
- flowspy: core application. Installation is done at /srv/flowspy::
80

    
81
    cd /srv
82
    git clone https://code.grnet.gr/git/flowspy
83
    cd flowspy
84

    
85
Application configuration
86
=========================
87
Copy settings.py.dist to settings.py::
88
    
89
    cd flowspy
90
    cp settings.py.dist settings.py
91

    
92
Edit settings.py file and set the following according to your configuration::
93

    
94
    ADMINS: set your admin name and email (assuming that your server can send notifications)
95
    DATABASES (to point to your local database). You could use views instead of tables for models: peer, peercontacts, peernetworks. For this to work we suggest MySQL with MyISAM db engine
96
    SECRET_KEY : Make this unique, and don't share it with anybody
97
    STATIC_ROOT: /srv/flowspy/static (or your installation directory)
98
    STATIC_URL (static media directory) . If you have followed the above this should be: /srv/flowspy/static
99
    TEMPLATE_DIRS : If you have followed the above this should be: /srv/flowspy/templates
100
    CACHE_BACKEND:  Enable Memcached for production or leave to DummyCache for development environments
101
    Alternatively you could go for redis with the corresponding Django client lib.
102
    NETCONF_DEVICE (tested with Juniper EX4200 but any BGP enabled Juniper should work). This is the flowspec capable device
103
    NETCONF_USER (enable ssh and netconf on device)
104
    NETCONF_PASS
105
    If beanstalk is selected the following should be left intact.
106
    BROKER_HOST (beanstalk host)
107
    BROKER_PORT (beanstalk port)
108
    SERVER_EMAIL
109
    EMAIL_SUBJECT_PREFIX
110
    If beanstalk is selected the following should be left intact.
111
    BROKER_URL (beanstalk url)
112
    SHIB_AUTH_ENTITLEMENT (if you go for Shibboleth authentication)
113
    NOTIFY_ADMIN_MAILS (bcc mail addresses)
114
    PROTECTED_SUBNETS (subnets for which source or destination address will prevent rule creation and notify the NOTIFY_ADMIN_MAILS)
115
    The whois client is meant to be used in case you have inserted peers with their ASes in the peers table and wish to get network info for each one in an automated manner.
116
    PRIMARY_WHOIS
117
    ALTERNATE_WHOIS
118
    If you wish to deploy FoD with Shibboleth change the following attributes according to your setup:
119
    SHIB_AUTH_ENTITLEMENT = 'urn:mace'
120
    SHIB_ADMIN_DOMAIN = 'example.com'
121
    SHIB_LOGOUT_URL = 'https://example.com/Shibboleth.sso/Logout'
122
    SHIB_USERNAME = ['HTTP_EPPN']
123
    SHIB_MAIL = ['mail', 'HTTP_MAIL', 'HTTP_SHIB_INETORGPERSON_MAIL']
124
    SHIB_FIRSTNAME = ['HTTP_SHIB_INETORGPERSON_GIVENNAME']
125
    SHIB_LASTNAME = ['HTTP_SHIB_PERSON_SURNAME']
126
    SHIB_ENTITLEMENT = ['HTTP_SHIB_EP_ENTITLEMENT']
127

    
128
If you have not installed an outgoing mail server you can always use your own account (either corporate or gmail, hotmail ,etc) by adding the following lines in settings.py::
129

    
130
    EMAIL_USE_TLS = True #(or False)
131
    EMAIL_HOST = 'smtp.example.com'
132
    EMAIL_HOST_USER = 'username'
133
    EMAIL_HOST_PASSWORD = 'yourpassword'
134
    EMAIL_PORT = 587 #(outgoing)
135

    
136
It is strongly advised that you do not change the following to False values unless, you want to integrate FoD with you CRM or members database. This implies that you are able/have the rights to create database views between the two databases::
137

    
138
    PEER_MANAGED_TABLE = True
139
    PEER_RANGE_MANAGED_TABLE = True
140
    PEER_TECHC_MANAGED_TABLE = True   
141

    
142
By doing that the corresponding tables as defined in peers/models will not be created. As noted above, you have to create the views that the tables will rely on. 
143

    
144
.. note::
145
    Soon we will release a version with django-registration as a means to add users and Shibboleth will become an alternative
146

    
147
Let's move on with some copies and dir creations::
148

    
149
    mkdir /var/log/fod
150
    chown www-data.www-data /var/log/fod
151
    cp urls.py.dist urls.py
152
    cd ..
153

    
154
.. note::
155
    LOG_FILE_LOCATION in settings.py is set to **/var/log/fod**. Adjust the chown command above to your selected dir.
156

    
157
System configuration
158
====================
159
Apache operates as a gunicorn Proxy with WSGI and Shibboleth modules enabled.
160
Depending on the setup the apache configuration may vary::
161

    
162
    a2enmod rewrite
163
    a2enmod proxy
164
    a2enmod ssl
165
    a2enmod proxy_http
166

    
167
If shibboleth is to be used::
168

    
169
    apt-get install libapache2-mod-shib2
170
    a2enmod shib2
171

    
172
Now it is time to configure beanstalk, gunicorn, celery and apache.
173

    
174
beanstalkd
175
----------
176
Enable beanstalk by editting /etc/default/beanstalkd::
177

    
178
    vim /etc/default/beanstalkd
179

    
180
Uncomment the line **START=yes** to enable beanstalk
181

    
182
Start beanstalkd::
183

    
184
    service beanstalkd start
185

    
186
gunicorn.d
187
----------
188
Create and edit /etc/gunicorn.d/fod::
189

    
190
    vim /etc/gunicorn.d/fod
191

    
192
FoD is served via gunicorn and is then proxied by Apache. If the above directory conventions have been followed so far, then your configuration should be::
193

    
194
    CONFIG = {
195
          'mode': 'django',
196
          'working_dir': '/srv/flowspy',
197
          'args': (
198
               '--bind=127.0.0.1:8081',
199
               '--workers=1',
200
               '--worker-class=egg:gunicorn#gevent',
201
               '--timeout=30',
202
               '--debug',
203
               '--log-level=debug',
204
               '--log-file=/var/log/gunicorn/fod.log',
205
          ),
206
    }
207

    
208

    
209
celeryd
210
-------
211
Celery is used over beanstalkd to apply firewall rules in a serial manner so that locks are avoided on the flowspec capable device. In our setup celery runs via django. That is why the python-django-celery package was installed.
212

    
213
Create the celeryd daemon at /etc/init.d/celeryd **if it does not already exist**::
214

    
215
    vim /etc/init.d/celeryd
216

    
217
The configuration should be::
218

    
219
    #!/bin/sh -e
220
    # ============================================
221
    #  celeryd - Starts the Celery worker daemon.
222
    # ============================================
223
    #
224
    # :Usage: /etc/init.d/celeryd {start|stop|force-reload|restart|try-restart|status}
225
    # :Configuration file: /etc/default/celeryd
226
    #
227
    # See http://docs.celeryq.org/en/latest/cookbook/daemonizing.html#init-script-celeryd
228
    
229
    
230
    ### BEGIN INIT INFO
231
    # Provides:              celeryd
232
    # Required-Start:     $network $local_fs $remote_fs
233
    # Required-Stop:       $network $local_fs $remote_fs
234
    # Default-Start:       2 3 4 5
235
    # Default-Stop:        0 1 6
236
    # Short-Description: celery task worker daemon
237
    # Description:          Starts the Celery worker daemon for a single project.
238
    ### END INIT INFO
239
    
240
    #set -e
241
    
242
    DEFAULT_PID_FILE="/var/run/celery/%n.pid"
243
    DEFAULT_LOG_FILE="/var/log/celery/%n.log"
244
    DEFAULT_LOG_LEVEL="INFO"
245
    DEFAULT_NODES="celery"
246
    DEFAULT_CELERYD="-m celery.bin.celeryd_detach"
247
    ENABLED="false"
248
    
249
    [ -r "$CELERY_DEFAULTS" ] && . "$CELERY_DEFAULTS"
250
    
251
    [ -r /etc/default/celeryd ] && . /etc/default/celeryd
252
    
253
    if [ "$ENABLED" != "true" ]; then
254
          echo "celery daemon disabled - see /etc/default/celeryd."
255
          exit 0
256
    fi
257
    
258
    
259
    CELERYD_PID_FILE=${CELERYD_PID_FILE:-${CELERYD_PIDFILE:-$DEFAULT_PID_FILE}}
260
    CELERYD_LOG_FILE=${CELERYD_LOG_FILE:-${CELERYD_LOGFILE:-$DEFAULT_LOG_FILE}}
261
    CELERYD_LOG_LEVEL=${CELERYD_LOG_LEVEL:-${CELERYD_LOGLEVEL:-$DEFAULT_LOG_LEVEL}}
262
    CELERYD_MULTI=${CELERYD_MULTI:-"celeryd-multi"}
263
    CELERYD=${CELERYD:-$DEFAULT_CELERYD}
264
    CELERYCTL=${CELERYCTL:="celeryctl"}
265
    CELERYD_NODES=${CELERYD_NODES:-$DEFAULT_NODES}
266
    
267
    export CELERY_LOADER
268
    
269
    if [ -n "$2" ]; then
270
          CELERYD_OPTS="$CELERYD_OPTS $2"
271
    fi
272
    
273
    CELERYD_LOG_DIR=`dirname $CELERYD_LOG_FILE`
274
    CELERYD_PID_DIR=`dirname $CELERYD_PID_FILE`
275
    if [ ! -d "$CELERYD_LOG_DIR" ]; then
276
          mkdir -p $CELERYD_LOG_DIR
277
    fi
278
    if [ ! -d "$CELERYD_PID_DIR" ]; then
279
          mkdir -p $CELERYD_PID_DIR
280
    fi
281
    
282
    # Extra start-stop-daemon options, like user/group.
283
    if [ -n "$CELERYD_USER" ]; then
284
          DAEMON_OPTS="$DAEMON_OPTS --uid=$CELERYD_USER"
285
          chown "$CELERYD_USER" $CELERYD_LOG_DIR $CELERYD_PID_DIR
286
    fi
287
    if [ -n "$CELERYD_GROUP" ]; then
288
          DAEMON_OPTS="$DAEMON_OPTS --gid=$CELERYD_GROUP"
289
          chgrp "$CELERYD_GROUP" $CELERYD_LOG_DIR $CELERYD_PID_DIR
290
    fi
291
    
292
    if [ -n "$CELERYD_CHDIR" ]; then
293
          DAEMON_OPTS="$DAEMON_OPTS --workdir=\"$CELERYD_CHDIR\""
294
    fi
295
    
296
    
297
    check_dev_null() {
298
          if [ ! -c /dev/null ]; then
299
               echo "/dev/null is not a character device!"
300
               exit 1
301
          fi
302
    }
303
    
304
    
305
    export PATH="${PATH:+$PATH:}/usr/sbin:/sbin"
306
    
307
    
308
    stop_workers () {
309
          $CELERYD_MULTI stop $CELERYD_NODES --pidfile="$CELERYD_PID_FILE"
310
    }
311
    
312
    
313
    start_workers () {
314
          $CELERYD_MULTI start $CELERYD_NODES $DAEMON_OPTS           \
315
                                        --pidfile="$CELERYD_PID_FILE"        \
316
                                        --logfile="$CELERYD_LOG_FILE"        \
317
                                        --loglevel="$CELERYD_LOG_LEVEL"     \
318
                                        --cmd="$CELERYD"                           \
319
                                        $CELERYD_OPTS
320
    }
321
    
322
    
323
    restart_workers () {
324
          $CELERYD_MULTI restart $CELERYD_NODES $DAEMON_OPTS        \
325
                                           --pidfile="$CELERYD_PID_FILE"     \
326
                                           --logfile="$CELERYD_LOG_FILE"     \
327
                                           --loglevel="$CELERYD_LOG_LEVEL"  \
328
                                           --cmd="$CELERYD"                        \
329
                                           $CELERYD_OPTS
330
    }
331
    
332
    
333
    
334
    case "$1" in
335
          start)
336
               check_dev_null
337
               start_workers
338
          ;;
339
    
340
          stop)
341
               check_dev_null
342
               stop_workers
343
          ;;
344
    
345
          reload|force-reload)
346
               echo "Use restart"
347
          ;;
348
    
349
          status)
350
               $CELERYCTL status $CELERYCTL_OPTS
351
          ;;
352
    
353
          restart)
354
               check_dev_null
355
               restart_workers
356
          ;;
357
    
358
          try-restart)
359
               check_dev_null
360
               restart_workers
361
          ;;
362
    
363
          *)
364
               echo "Usage: /etc/init.d/celeryd {start|stop|restart|try-restart|kill}"
365
               exit 1
366
          ;;
367
    esac
368
    
369
    exit 0
370

    
371
celeryd configuration
372
---------------------
373
celeryd requires a /etc/default/celeryd file to be in place.
374
Thus we are going to create this file (/etc/default/celeryd)::
375

    
376
    vim /etc/default/celeryd
377

    
378
Again if the directory conventions have been followed the file is (pay attention to the CELERYD_USER, CELERYD_GROUP and change accordingly)  ::
379

    
380
    # Default: false
381
    ENABLED="true"
382
    
383
    # Name of nodes to start, here we have a single node
384
    CELERYD_NODES="w1"
385
    # or we could have three nodes:
386
    #CELERYD_NODES="w1 w2 w3"
387
    
388
    # Where to chdir at start.
389
    CELERYD_CHDIR="/srv/flowspy"
390
    # How to call "manage.py celeryd_multi"
391
    CELERYD_MULTI="python $CELERYD_CHDIR/manage.py celeryd_multi"
392
    
393
    # How to call "manage.py celeryctl"
394
    CELERYCTL="python $CELERYD_CHDIR/manage.py celeryctl"
395
    
396
    # Extra arguments to celeryd
397
    #CELERYD_OPTS="--time-limit=300 --concurrency=8"
398
    CELERYD_OPTS="-E -B --schedule=/var/run/celery/celerybeat-schedule --concurrency=1 --soft-time-limit=180 --time-limit=1800"
399
    # Name of the celery config module.
400
    CELERY_CONFIG_MODULE="celeryconfig"
401
    
402
    # %n will be replaced with the nodename.
403
    CELERYD_LOG_FILE="/var/log/celery/fod_%n.log"
404
    CELERYD_PID_FILE="/var/run/celery/%n.pid"
405
    
406
    CELERYD_USER="root"
407
    CELERYD_GROUP="root"
408
    
409
    # Name of the projects settings module.
410
    export DJANGO_SETTINGS_MODULE="flowspy.settings"
411

    
412
Apache
413
------
414
Apache proxies gunicorn. Things are more flexible here as you may follow your own configuration and conventions. Create and edit /etc/apache2/sites-available/fod. You should set <server_name> and <admin_mail> along with your certificates. If under testing environment, you can use the provided snakeoil certs. If you do not intent to use Shibboleth delete or comment the corresponding configuration parts inside **Shibboleth configuration** ::
415

    
416
    vim /etc/apache2/sites-available/fod
417

    
418
Again if the directory conventions have been followed the file should be::
419

    
420
    <VirtualHost *:80>
421
        ServerAdmin webmaster@localhost
422
        ServerName  fod.example.com
423
        DocumentRoot /var/www
424
    
425
        ErrorLog ${APACHE_LOG_DIR}/fod_error.log
426
    
427
        # Possible values include: debug, info, notice, warn, error, crit,
428
        # alert, emerg.
429
        LogLevel debug
430
        
431
        CustomLog ${APACHE_LOG_DIR}/fod_access.log combined
432
    
433
        Alias /static       /srv/flowspy/static
434
          RewriteEngine On
435
          RewriteCond %{HTTPS} off
436
          RewriteRule ^/(.*) https://fod.example.com/$1 [L,R]
437
    </VirtualHost>
438
    
439
    <VirtualHost *:443>
440
        ServerName    fod.example.com
441
        ServerAdmin     webmaster@localhost
442
        ServerSignature        On
443
        
444
        SSLEngine on
445
        SSLCertificateFile    /etc/ssl/certs/fod.example.com.crt
446
        SSLCertificateChainFile /etc/ssl/certs/example-chain.pem
447
        SSLCertificateKeyFile    /etc/ssl/private/fod.example.com.key
448
    
449
        AddDefaultCharset UTF-8
450
        IndexOptions        +Charset=UTF-8
451
    
452
        ShibConfig       /etc/shibboleth/shibboleth2.xml
453
        Alias          /shibboleth-sp /usr/share/shibboleth
454
    
455
    
456
        <Location /login>
457
             AuthType shibboleth
458
             ShibRequireSession On
459
             ShibUseHeaders On
460
             ShibRequestSetting entityID https://idp.example.com/idp/shibboleth
461
             require valid-user
462
        </Location>
463
        
464
        # Shibboleth debugging CGI script
465
        ScriptAlias /shibboleth/test /usr/lib/cgi-bin/shibtest.cgi
466
        <Location /shibboleth/test>
467
             AuthType shibboleth
468
             ShibRequireSession On
469
             ShibUseHeaders On
470
             require valid-user
471
        </Location>
472
    
473
        <Location /Shibboleth.sso>
474
             SetHandler shib
475
        </Location>
476
    
477
        # Shibboleth SP configuration
478
    
479
        #SetEnv                       proxy-sendchunked
480
        
481
              <Proxy *>
482
               Order allow,deny
483
               Allow from all
484
               </Proxy>
485
    
486
               SSLProxyEngine           off
487
               ProxyErrorOverride     off
488
          ProxyTimeout     28800
489
             ProxyPass        /static !
490
             ProxyPass          /shibboleth !
491
             ProxyPass        /Shibboleth.sso !
492
             
493
               ProxyPass           / http://localhost:8081/ retry=0
494
               ProxyPassReverse / http://localhost:8081/
495
    
496
          Alias /static          /srv/flowspy/static
497
    
498
        LogLevel warn
499
        
500
        ErrorLog ${APACHE_LOG_DIR}/fod_error.log
501
          CustomLog ${APACHE_LOG_DIR}/fod_access.log combined
502
    
503
    </VirtualHost>
504

    
505
Now, enable your site. You might want to disable the default site if fod is the only site you host on your server::
506

    
507
    a2dissite default
508
    a2ensite fod
509

    
510
You are not far away from deploying FoD. When asked for a super user, create one::
511

    
512
    cd /srv/flowspy
513
    python manage.py syncdb
514
    python manage.py migrate longerusername
515
    python manage.py migrate flowspec
516
    python manage.py migrate djcelery
517
    python manage.py migrate accounts
518

    
519
If you have not changed the values of the PEER\_\*\_TABLE variables to False and thus you are going for a default installation (that is PEER\_\*\_TABLE variables are set to True) , then run::
520
    
521
    python manage.py migrate peers
522

    
523
If however you have set the PEER\_\*\_TABLE variables to False and by accident you have ran the command above, then you have to cleanup you database manually by dropping the peer\* tables plus the techc_email table. For MySQL the command is::
524
    
525
    DROP TABLE `peer`, `peer_networks`, `peer_range`, `peer_techc_emails`, techc_email;  
526

    
527
Restart, gunicorn and apache::
528

    
529
    service gunicorn restart && service apache2 restart
530

    
531

    
532
Propagate the flatpages
533
=======================
534
Inside the initial_data/fixtures_manual.xml file we have placed 4 flatpages (2 for Greek, 2 for English) with Information and Terms of Service about the service. 
535
To import the flatpages, run from root folder::
536

    
537
    python manage.py loaddata initial_data/fixtures_manual.xml
538

    
539

    
540

    
541
Testing the platform
542
====================
543
Log in to the admin interface via https:\/\/<hostname>\/admin. Go to Peer ranges and add a new range (part of/or a complete subnet), eg. 10.20.0.0/19
544
Go to Peers and add a new peer, eg. id: 1, name: Test, AS: 16503, tag: TEST and move the network you have created from Avalable to Chosen. From the admin front, go to User, and edit your user. From the bottom of the page, select the TEST peer and save.
545
Last but not least, modify as required the existing (example.com) Site instance (admin home->Sites). You are done. As you are logged-in via the admin, there is no need to go through Shibboleth at this time. Go to https:\/\/<hostname>\/ and create a new rule. Your rule should be applied on the flowspec capable device after aprox. 10 seconds. If no Shibboleth authentication is available, a https:\/\/<hostname>\/altlogin is provided.
546

    
547
Branding
548
========
549
Via the admin interface you can modify flatpages to suit your needs
550

    
551
Footer
552
------
553
Under the templates folder (templates), you can alter the footer.html file to include your own footer messages, badges, etc.
554

    
555
Welcome Page
556
------------
557
Under the templates folder (templates), you can alter the welcome page - welcome.html with your own images, carousel, videos, etc.