kiskadee development¶
This section is a guide for new developers willing to set up a development environemnt to start contributing to kiskadee development. If you have any doubt, please contact us on IRC in #kiskadee at freenode.
kiskadee development only suports python versions >= 3. Assume such versions for all commands run along this documentation.
Installing dependencies¶
The name of the dependencies are compatible with the Fedora distribution. If you use another OS, you will have to find the compatible names for the dependencies.
dnf is Fedora package manager. One should make proper subtitutions when using another linux distribution.
The redhat-rpm-config package, is a specific Fedora dependency, if you do not use Fedora (or a Red Hat distribution), you may not need to install it.
# dnf install gcc openssl-devel python3-devel python3-pip redhat-rpm-config
Python dependencies¶
To handle python dependencies, we recommend installing a virtual environment as shown in the subsections below. Regardless of how you handle your python pypi dependencies, just run
$ pip install -r requirements.txt
to install all kiskadee python related dependencies.
Creating a python virtual environment with virtualenv¶
Using virtualenvwrapper¶
# dnf install python3-virtualenvwrapper python3-virtualenv
After installing the packages above, it is necessary to export some environment variables.
$ export WORKON_HOME=$HOME/.virtualenvs
$ source /usr/bin/virtualenvwrapper.sh
Note that you may want to add the lines above in your .bashrc file.
Then just execute
$ mkvirtualenv kiskadee -p python3
$ workon kiskadee
Using virtualenv¶
# dnf install python3-virtualenvwrapper python3-virtualenv
$ virtualenv-3 .venv
$ source bin/activate
You may need to change the version of the binary file for virtualenv-3.6 or whatever minor python version you use in your system
Docker Images¶
To run the static analyzers, you must have Docker installed and running. If you have configured the Docker engine properly, run the analyzers target in the Makefile. It will build the images for you.
$ make analyzers
Database¶
Now we will create the kiskadee database. You will need to install the postgresql packages for your system. If you use Fedora, follow the next steps, if not, you will have to find out how install postgresql on your system.
# dnf install postgresql-server postgresql-contrib
# systemctl enable postgresql
# postgresql-setup initdb
# systemctl start postgresql
To install on distributions different from Fedora, refer to their documentation.
With postgresql installed, you will need to create the kiskadee role and database.
# su - postgres
$ createdb kiskadee
$ createdb kiskadee_test
$ createuser kiskadee -P
$ # here, we use kiskadee as password.
$ psql -U postgres -c "grant all privileges on database kiskadee to kiskadee"
$ psql -U postgres -c "grant all privileges on database kiskadee_test to kiskadee"
# go back to your user (ctrl+d)
$ echo "localhost:5432:kiskadee:kiskadee:kiskadee" > ~/.pgpass
$ chmod 600 ~/.pgpass
Restart the postgresql service:
# systemctl restart postgresql
Test the database connection:
$ psql -U kiskadee -d kiskadee
If you were not able to log in on the database, you will need to edit the pg_hba.conf and change some rules defined by the postgresql package. On Linux systems this file normally stays at the /var/lib/pgsql/data/. Open this file and change it to:
# "local" is for Unix domain socket connections only
local all all md5
# IPv4 local connections:
host all all 127.0.0.1/32 md5
# IPv6 local connections:
host all all ::1/128 md5
After this change, restart postgresql service:
# systemctl restart postgresql
Test the database connection:
# psql -U kiskadee -d kiskadee
If you were able to log in the psql shell, the database is properly configured. Leave the shell with ctrl+d.
Migrations¶
Kiskadee uses alembic as its tool for database migration, it has a solid documentation on: http://alembic.zzzcomputing.com/en/latest
For short, the most used commands are:
To create a new migration
$ alembic revision -m "migration description"
To autogenerate a new migration
$ alembic revision --autogenerate
or
$ alembic revision --autogenerate -m "some migration description"
To execute the migrations
$ alembic upgrade head
$ alembic upgrade +2
$ alembic upgrade -1
$ alembic upgrade some_revision_id+2
Downgrading
$ alembic downgrade base
Environment variables¶
Kiskadee database migration tool(alembic) get its database configuration from a environment variable named DATABASE_TYPE. If this variable is not defined, then it will assume its running on a developemnt environment, but for others environments such as test, homologation or production be sure to set which one are being used.
Only set DATABASE_TYPE if kiskadee is running on a non development environment.
$ export DATABASE_TYPE=db_test
To see which data each one of those alembic will use, check util/kiskadee.conf
Running your first analysis¶
kiskadee reads environment variables from the util/kiskadee.conf file. Open the kiskadee.conf file, and set the example_fetcher as active (active = yes), the other fetchers will stay inactive (active = no).
Now run kiskadee by typing kiskadee on the terminal (kiskadee must be installed through python setup.py install). If the Docker images were properly built and the Docker client was properly configured on your machine, kiskadee will be able to analysis an example source code. This code is in the kiskadee/tests/test_source/ directory.
kiskadee will decompress the example source, and run the analyzers defined on the kiskadee.conf file. You can use any postgresql client to access the database that you have created and check the analysis performed by kiskadee.
## Fetchers
### Debian Fetcher If you intend to use the debian fetcher, you will have to install the devscripts package to use the necessary Debian tools to run the fetcher.
### Anitya Fetcher If you intend to run the anitya fetcher, you will have to install fedmsg-hub to enable kiskadee to consume the fedmsg events. To install fedmsg-hub run these commands in kiskadee root path:
# Run this inside the kiskadee's virtualenv
# mkdir -p /etc/fedmsg.d/
# cp util/base.py util/endpoints.py /etc/fedmsg.d/
# cp util/anityaconsumer.py /etc/fedmsg.d/
$ PYTHONPATH=`pwd` fedmsg-hub
With this steps, fedmsg-hub will instantiate AnityaConsumer and publish the monitored events using ZeroMQ. When kiskadee starts, it will consume the messages published by the consumer, and will run the analysis.
The events that comes to the anitya fetcher are published by Anitya, on this page. For more info about the Anitya service, read the rest of kiskadee documentation.
Tests and coverage¶
To check kiskadee tests and coverage just run:
$ make check
To check kiskadee coverage open the file covhtml/index.html.
building docs¶
To build the documentation just run
$ cd doc
$ make check
Note that you must be in the doc directory
To access the documentation open the index.html file, inside doc/_build/html.