PyCharm

The Python IDE for data science and web development

Download

Tips & Tricks Tutorials

Developing in a VM with Vagrant and Ansible

Ernst Haagsman

One of the things that could make developing cloud applications hard, would be differences between the dev environment and the production environment. This is why one of the factors of the twelve factor app is maintaining dev-prod parity. Today we’ll start a blog series about developing cloud applications, and we’ll discuss how to set up a local development environment using Vagrant.

We’ll use these technologies for this application:

Vagrant
Ansible
Flask
Virtualenv
Ubuntu

Today we’ll just create a simple Flask application that’ll say ‘Hello world’. In the next post in this series, we’ll introduce a larger application that we’ll deploy to AWS in a future post.

If you want to follow along at home, you can find the code from today’s blog post on GitHub. See the commit history there to see the progress from the beginning to the end.

Getting Started

So let’s create a project, and get started. If you want to follow along, you’ll need to have Vagrant, Virtualbox, and PyCharm Professional Edition installed on your computer.

Open PyCharm, and create a new pure Python project.

The first step will be to set up the Vagrant VM, and configure the necessary items. In the project folder, run vagrant init -m bento/ubuntu-16.04. You can run commands within PyCharm by opening the terminal (Alt + F12).

This generates a Vagrantfile that only contains the base box that we’re using. If we run vagrant up at this point, we’d get a plain Ubuntu server box. For our project we’ll need to install some things and expose some ports though, so let’s add this to the Vagrantfile:

Vagrant.configure("2") do |config|
 config.vm.box = "bento/ubuntu-16.04"

 config.vm.network "forwarded_port", guest: 5000, host: 5000

 config.vm.provision "ansible_local" do |a|
   a.playbook = "setup.yml"
 end
end

The ansible_local provisioner will install Ansible on the Ubuntu VM and then run it there, this means we don’t need to install Ansible on our host computer. Ansible lets us describe the desired state for a computer, and will then make the necessary changes to achieve that state. So let’s have a look at what’s necessary to install Python 3.6 on the VM.

Provisioning a VM with Ansible

Ansible works with Playbooks. These are YAML files that describe what state should be applied to what machines. Let’s create setup.yml, and try to install Python 3.6:

---
- hosts: all
  become: yes # This means that all tasks will be executed with sudo
  tasks:
   - name: Install Python 3.6
     apt:
       name: python3.6
       state: present
       update_cache: yes

A playbook is a list of plays on the top level. We can configure per play which hosts we want to apply it to, whether we need to become another user, and a list of tasks. In our example, we apply the play to all hosts: there’s only one host in the Vagrant setup, so that’s easy enough. We also set become to yes, which has the effect of running our tasks with sudo.

The tasks are the way we can configure the desired state of our VM. We can name our tasks to make it easier for us to see what’s going on, but Ansible doesn’t technically need it. The task we have here is just an instruction for Ansible to use the apt module, which is bundled with Ansible. We specify three options to the apt module:

The name of the package we’re interested in
The state we’d like the package to be in: present on the machine
Update the apt cache before installing

This last option basically means that Ansible will run apt update before running apt install, if necessary.

If you’re thinking, isn’t this just a very hard way to write sudo apt update && sudo apt install python3.6, at this point you’re right. However, the value of Ansible is that you’re not describing actions, but you’re describing a desired state. So the second time you run Ansible, it detects Python 3.6 is already installed, and it won’t do anything. Idempotence is one of Ansible’s core principles. Another key benefit is that you can version control changes to server configuration.

So let’s run vagrant up (Ctrl+Shift+A to Find action, and then type vagrant up), and we should have a VM with Python 3.6!

Trouble in Paradise

TASK [Install Python 3.6] ******************************************************

fatal: [default]: FAILED! => {"changed": false, "msg": "No package matching 'python3.6' is available"}

       to retry, use: --limit @/vagrant/setup.retry

Unfortunately, Python 3.6 isn’t available from Ubuntu’s default package repositories. There are several ways to resolve this situation, the easiest would be to find a PPA (Personal Package Archive) which has Python 3.6.

A PPA which is mentioned in many places on the internet is Jonathon F’s PPA. So how would we go about adding this PPA using Ansible? Turns out there are two modules that can help us out here, apt_key and apt_repository. Apt_key allows us to specify the public key associated with the repository, to make sure any releases we get are really from Jonathon. And apt_repository then adds the repository to the apt configuration. So let’s add these two tasks to the playbook, before the install task (Ansible runs tasks in the order specified):

- name: Add key for jonathonf PPA
  apt_key:
    keyserver: keyserver.ubuntu.com
    id: 4AB0F789CBA31744CC7DA76A8CF63AD3F06FC659
    state: present

- name: Add jonathonf PPA
  apt_repository:
    repo: deb http://ppa.launchpad.net/jonathonf/python-3.6/ubuntu xenial main
    state: present

Now run vagrant provision (or Tools | Vagrant | Provision), to rerun the playbook. After completing, we should see the summary:

PLAY RECAP *********************************************************************
default                    : ok=4    changed=3    unreachable=0    failed=0

At this point, let’s create a requirements.txt with the libraries we’ll use today, in this case, just Flask:

Flask==0.12

Most Linux distributions use the system interpreter themselves, that’s one of the reasons for virtualenvs being best practice. So let’s create a virtualenv, and then install these packages. As the python-3.6 package didn’t include pip, we’ll first need to install pip. Then, using pip, we’ll need to install virtualenv into the system interpreter. After that we’ll be able to create a new virtualenv with the requirements we specify. To do this, specify at the end of the playbook:

- name: Install pip3
  apt:
    name: python3-pip
    state: present
    update_cache: yes

- name: Install 'virtualenv' package
  pip:
    name: virtualenv
    executable: pip3

- name: Create virtualenv
  become: no
  pip:
    virtualenv: "/home/vagrant/venv"
    virtualenv_python: python3.6
    requirements: "/vagrant/requirements.txt"

First, we’re using the apt module to install pip. Then, we’re using Ansible’s pip module to install the virtualenv package. And finally we’re using the pip module again to now create the virtualenv, and then install the packages in the newly created virtualenv. Vagrant automatically mounts the project directory in the /vagrant folder in the VM, so we can refer to our requirements.txt file this way.

At this point we have our Python environment ready, and we could continue going the same way to add a database and anything else we might desire. Let’s have a look to see how we can organize our playbook further. Firstly, we’ve now hardcoded paths with ‘vagrant’, which prevents us from reusing the same playbook later on AWS. Let’s change this:

---
- hosts: all
  become: yes # This means that all tasks will be executed with sudo
  vars:
    venv_path: "/home/vagrant/venv"
    requirements_path: "/vagrant/requirements.txt"
 tasks:
… snip … 
  - name: Create virtualenv
    become: no
    pip:
      virtualenv: "{{ venv_path }}"
      virtualenv_python: python3.6
      requirements: "{{ requirements_path }}"

The first thing we can do is define variables for these paths. If the variable syntax looks familiar, that’s because it is: Ansible is written in Python, and uses jinja2 for templating.

If we were to add database plays to the same playbook, we’re mixing things that we may want to separate later. Wouldn’t it be easier to have these Python plays somewhere we can call them, and have the database plays in another place? This is possible using Ansible roles. Let’s refactor this playbook into a Python role.

Ansible roles are essentially a folder structure with YAML files that are used to specify the things necessary for the role. To refactor our plays into a Python role, we just need to create several folders: $PROJECT_HOME/roles/python/tasks, and then place a file called main.yml in that last tasks folder. Copy the list of tasks from our playbook into that file, making sure to unindent them:

- name: Add key for jonathanf PPA
  apt_key:
    keyserver: keyserver.ubuntu.com
    id: 4AB0F789CBA31744CC7DA76A8CF63AD3F06FC659
    state: present

... etc ...

Afterwards, specify in the playbook which role to apply:

---
- hosts: all
  become: yes # This means that all tasks will be executed with sudo
  vars:
    venv_path: "/home/vagrant/venv"
    requirements_path: "/vagrant/requirements.txt"
  roles:
    - {role: python}

That’s all there’s to it! To make sure everything runs smoothly still, run vagrant provision once more to make sure everything is applied to the VM.

Running Code from PyCharm

Now that we have a provisioned VM ready to go, let’s write some code!

First let’s set up the Python interpreter. Go to File | Settings | Project Interpreter. Then use the gear icon to select ‘Add Remote’, and choose Vagrant. PyCharm automatically detects most settings, we just need to put the path to the Python interpreter to tell PyCharm about the virtualenv we created:

Now create a new script, let’s name it server.py and add Flask’s Hello World:

from flask import Flask
app = Flask(__name__)

@app.route("/")
def hello():
    return "Hello World!"

if __name__ == '__main__':
    app.run(host='0.0.0.0', debug=True)

Make sure that you use the host='0.0.0.0' kwarg, as Flask by default only binds to localhost, and we wouldn’t be able to access our application later.

Now to create a run configuration, just navigate to the script as usual, and select ‘Single instance only’ to prevent the app not starting when the port is already in use:

By marking the run configuration as ‘single instance only’ we make sure that we can’t accidentally start the script twice and get a ‘Port already in use’ error.

After saving the run configuration, just click the regular Run or Debug button, and the script should start.