ODM Wiki

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
FIXME **This page is outdated. Please help completing the wiki updates.**\\ //(remove this paragraph once the page is updated)//

====== Setting up test & development environment ======

===== Who is this guide for =====

  * Developers contributing on the development of ODM's modules
  * ODM Staff testing the actual development 
  * System administrators willing to understand better ODM'S architecture

===== What this guide teaches =====

  * How to replicate the test and development environment using both Vagrant and Webfaction.

===== Things to know in forehand =====

==== What is Vagrant ====

[[http://www.vagrantup.com|Vagrant]] is free and open-source software for creating and configuring virtual development environments. It can be seen as a wrapper around virtualization software such as VirtualBox, and around configuration management software such as Chef, Salt or Puppet.

==== What is Ansible ====

[[http://www.ansible.com|Ansible]] is an open-source software platform for configuring and managing computers. It combines multi-node software deployment, ad hoc task execution, and configuration management. It manages nodes over SSH and requires Python (2.4 or later) to be installed on them. Modules work over JSON and standard output and can be written in any programming language. The system uses YAML to express reusable descriptions of systems.

===== Using Virtualized environment in local machine (Vagrant) =====

ODM developers can choose Vagrant for managing Virtual Machines where testing and development tasks can be undertaken. For this, following software needs to be installed and configured:

  * [[http://vagrantup.com/|Vagrant]]
  * [[https://www.virtualbox.org/|Virtual Box]]

==== Provisioning a Vagrant VM with Ansible ====

=== Step 1: get deployment scripts ===

[[https://github.com/OpenDevelopmentMekong/odm-deployment|odm-deployment]] repo on github contains, among other scripts, the necessary files to configure and provision a Vagrant box for development. 
For cloning it using the terminal, type:

<code>git clone https://github.com/OpenDevelopmentMekong/odm-deployment.git</code>

This command will download the code from the repository, it contains also other useful scripts that we will need afterwards. Following files interests us particularly: 

  * **odm-deployment/Vagrantfile** 
  * **odm-deployment/bootstrap.sh**
  * **odm-deployment/ansible/***

=== Step 2: Edit the configuration files ===

  * **odm-deployment/ansible/group_vars/webfaction-servers.sample**: This file needs to be renamed to **webfaction-servers** and be modified according to paths, urls and port numbers for deploying on a webfaction server.
  * **odm-deployment/ansible/group_vars/vagrant-box.sample**: This file needs to be renamed to **vagrant-box** and be modified according to paths, urls and port numbers for deploying on the vagrant machine. 
      * Below are some settings that needs to provide, open the **vagrant-box** and provide the information of: 
        * tomcat_admin_user:
        * tomcat_admin_password:
        * datapusher_port:
        * wp_db_name:  
        * wp_db_user:  
        * wp_db_password:  
        * wp_domain:   

  * **odm-deployment/ansible/roles**: This file contains ansible playbooks for the different components of the architecture. 
  * **odm-deployment/ansible/stage.sample**: This file needs to be renamed to **stage** is the ansible inventory and needs to be configured with URLs to the machines to provision. 
 

=== Step 3: Starting the box for the first time provisions it ===

In order to start the development box, change directory to where the Vagrantfile is and type: 

<code>
cd odm-deployment
vagrant up   
</code>

This will take some time because the virtual box will be provisioned with the instructions specified in the bootstrap.sh. This script basically install some needed dependencies such as git and ansible and runs the set of Ansible instructions available on deployment/ansible in order to provision the VM. 

The provisioning process might take between 5 and 15 minutes, depending on the power of your machine and speed of your internet connection. 

<WRAP center round tip 90%>
For more information about provisioning Vagrant, refer to [[https://docs.vagrantup.com/v2/provisioning/index.html|Vagrant documentation]]
</WRAP>

Aparently, the VM needs to be restarted once after the deployment is ready in order for the different services to start properly. Following command needs to be run:
<code>
vagrant reload
</code>

==== If Errors occur during vagrant up ====

Sometimes, the deployment of the Development Environment will be stuck at some points that caused by wrong configuration or the speed of internet connection.

=== Double Check the Configuration ===

* Please check "ansible\group_vars\**vagrant-box**" to make sure that all requirement attributes below are provided: 
        * tomcat_admin_user:
        * tomcat_admin_password:
        * datapusher_port:
        * wp_db_name:  
        * wp_db_user:  
        * wp_db_password:  
        * wp_domain:  

=== Change the location of Ubuntu servers ===

* If they are correctly provided, then try to change the location of Ubuntu servers to different regions that is closer to your location. 
  * Unordered List ItemGo to "ansible/roles/common-ubuntu/files", you will see a series of .list files that contain Ubuntu servers from different regions. 
  * Then update the source file name in **main.yml** located in "ansible\roles\common-ubuntu\tasks".
  * For example, in line 3:   
The default is "copy: src=sources.list dest=/etc/apt/sources.list owner=root group=root mode=0644" 
updated to: "copy: src=sources.th.list dest=/etc/apt/sources.list owner=root group=root mode=0644" 
  * Next vagrant destroy and up again. 

=== Fail due to Internet connection issue ===

Please go ahead to download the solr and tomcat below and place them in "odm-deployment\installables".

  * http://archive.apache.org/dist/lucene/solr/4.7.2/solr-4.7.2.tgz 
  * http://archive.apache.org/dist/tomcat/tomcat-7/v7.0.56/bin/apache-tomcat-7.0.56.tar.gz
  * Rename them to : **solr.tgz** and **tomcat.tar.gz**
  * Then vagrant destroy and up again. 
  * Done

=== Log in with ssh once provisioning finish. ===

Once the process is finished and no errors have occurred, it is time to access the machine via ssh. There are still some necessary steps to be done.

<code>
vagrant ssh  
</code>

=== Stopping the box ===

Exit the ssh session and call vagrant halt, this way the box will be stopped, you can start it any time by calling vagrant up again.

<code>
exit
vagrant halt
</code>

===== Deploying on Webfaction with Ansible =====

Replicating the system on a Webfaction instance uses the same Ansible scripts used for the Vagrant VM. 

=== Step 1: Edit configuration deployment scripts ===

As specified above, the **ansible/group_vars/webfaction-servers** file must be edited with URLs and Ports defined on webfaction prior to starting the provisioning.

=== Step 2: Configure Domains, Databases and Apps on webfaction ===

{{ ::domain_list_-_webfaction_control_panel.png?direct&600 |}}

Add a new domain such as ckan.yourwebfactionusername.webfactional.com and press save. For the sake of the example, my webfaction user is opendevmk therefor it'll be ckan.opendevmk.webfactional.com

Now we need to add applications for Tomcat and Ckan, which basically reserves a port on the server and optionally exposes it on the firewall or a domain. Go to "Applications" then click "Add new application"

{{ ::edit_application_ckan_-_webfaction_control_panel.png?direct&600 |}}

Repeat for app name ckan then when you click on the entry you've just added you'll see a port number. Note them down you'll need it later.

{{ ::add_new_application_-_webfaction_control_panel.png?direct&600 |}}

Lastly, click on dabase -> databases --> add new database and add the postgresql database for ckan.

{{ ::add_new_database_-_webfaction_control_panel.png?direct&600 |}}

=== Step 3: Run the ansible playbooks ===

In order to start the provisioning, we need to run the ansible-playbook command:

<code>
cd ansible
ansible-playbook site.yml -i stage 
</code>

===== After provisioning =====

=== Finish configuring CKAN ===

The provisioning scripts installs CKAN and its dependencies plus the [[https://github.com/OpenDevelopmentMekong/ckanext-odm_theme|odm_theme]] & [[https://github.com/OpenDevelopmentMekong/ckanext-odm_utils|odm_utils]] extensions, but there are still some actions to be taken in order to configure the development environment completely.

<WRAP center round info 90%>
Please refer to [[migrating_from_previous_system|Migrating from previous System]] and follow the instructions specified there in order to initialize the recently created test & development environment with real data.
</WRAP>

=== Finish configuring WP ===

The previous Ansible scripts provisions the box with a basic wordpress installation which will be listening to port 80. In order to finish configuring it point your browser to the URL specified on the configuration files and follow the instructions there. 

=== Accessing the code on the Development Box ===

There are several possibilities to access the files system on the VM in order to modify code files such as [[http://www.howtogeek.com/66776/how-to-remotely-copy-files-over-ssh-without-entering-your-password/|copying files over with SCP command]] or [[https://help.ubuntu.com/community/SettingUpNFSHowTo|creating a NFS mount point on the VM]].

===== Troubleshooting =====

=== Failed to fetch mirror error ===

Sometimes it can happen that the during the "Update packages list" step of the **common** Ansible playbook, the following error is thrown, stoping the execution of the provisioning.

{{ :mirror_not_found_error.png?600 |}}

In order to solve this, a failsafe sources list can be selected. For that, it is just necessary to change the following line in the **deployment/ansible/roles/common/tasks/main.yml** file:

<code>
- name: add new sources.list that checks for mirror
  copy: src=sources.list dest=/etc/apt/sources.list owner=root group=root mode=0644
  sudo: yes
</code>

And replace src=sources.list with one of the available failsafe sources lists in **deployment/ansible/roles/common/tasks/main.yml**

=== On start, CKAN and Tomcat not started ===

If the machine is provisioned and started but CKAN and/or Tomcat are not accesible through their respective URLs, it can be that **supervisor** has not being able to start them for some reason. In order to manually start them:

  - Activate virtualenv: <code>. ~/.virtualenvs/ckan/bin/activate</code>
  - Run supervisorctl: <code>supervisorctl</code> 
  - If following error message appears: //unix:///home/vagrant/tmp/supervisor.sock// no such file run: **supervisord**.
ODM Wiki

User Tools

Site Tools

Page Tools
