Au revoir Django forms

A few months ago, I released version 0.1.0 of django-rest-fomly, and I think that it’s time to speak about the project. First, I’ll speak about the reasons motivating me to start the project, then I’ll speak about the tool itself.

1. Why django-rest-formly ?

I start using Django since 2011, and I really like it, especially because it’s a real web framework and there is a great and active community behind it. From a design perspective, I like the fact that it emphasizes DRY and CoC principles. Besides, Django comes with powerful batteries making web development rapid, including forms, templates, admin site, ORM , etc. I like all these packages, but I think that some of them, like forms, aren’t useful anymore.  Nowadays, we speak about Single Page Application (SPA), web 2.0 and Internet of Thing (IoT). In such world, there is no need for Django forms package. Yep, there are a lot of packages developed for web 1.0 and the DSF community is aware of it. In fact, they’re acting according to this reality. On December 2015, Mozilla awarded $150,000 for rewriting Django to support WebSocket and to integrate key parts of Django REST Framework, among other things (more information here).

As a software engineer, I understand the importance of separating backend from frontend, and this is what I do. This is a good practice that you had to adopt if you aren’t. One of the advantages of this practice is the possibility to use the best tool for each part. Personally, I use AngularJS in the frontend, the Angular application communicates with the backend through a RESTful interface built using Django REST framework. However, I found myself writing by hand what Django forms was doing for me, and I didn’t like that, this is a waste of time. I spent a lot of time looking for a solution that I can use to solve this problem but I didn’t find any frontend package. So, I decided to create a generic solution to this use-case.

The idea was very simple. I need an Angular module which could replace Django forms package. In other words, this angular module should be able to create forms from a configuration object, add validation for fields, etc. Generally, I didn’t like the idea of building something from scratch, especially if I find something very powerful. In my case, it was the angular-formly module: It offers all that I expect and more, but the configuration structure is not compatible with the Django REST framework. angular-formly expects something like this:

{
    key: 'email',
    type: 'input',
    templateOptions: {
        type: 'email',
        label: 'Email address',
        placeholder: 'Enter email'
    }
}

below Django REST metadata for email field:

{
  "actions": {
    "POST": {
        "email": {
            "type": "email",
            "required": false,
            "read_only": false,
            "label": "Email address"
        }
    }
  }
}

Now, I think that the idea becomes clear. django-rest-formly project gives you MAINLY a CLI tool able to create an angular-formly form configuration object for any Django REST endpoint.

2. Installation

django-rest-formly is a CLI tool build on top of Node.js. So,  you can install it with npm (Node Package Manager) :

$ npm install -g django-rest-formly

3. How to use it

If you already installed the package, the django-rest-formly command should be available. django-rest-formly has two commands:

  • list: which list all available endpoints from the root API
  • form: which return the angular-formly form configuration for a given endpoint

Let’s suppose that we have a REST API at http://127.0.0.1:8000/api/v1/, I can list all existing endpoints with this command:

$ django-rest-formly list --host 127.0.0.1 --port 8000 --root /api/v1
Available endpoints:
  snippets: http://127.0.0.1:8000/snippets.json
  users: http://127.0.0.1:8000/users.json

To have an idea about the users’ endpoint, we had to use the form command:

$ django-rest-formly form --host 127.0.0.1 --port 8000 --root /api/v1 users

I think that it’ll be better that you interact with the tool. So I recommend you to follow the below steps:

1) clone the django-rest-framework-tutorial repository from my GitHub account: if you have git installed, just run

$ git clone http://github.com/benzid-wael/todo

2) Now, install create a python virtual environment and install the dependencies:

$ cd /path/to/django-rest-framework-tutorial
$ virtualenv --no-site-packages env
(env) $ pip install -r requirements.txt

3) Now, we can start the server

$ python manage.py runserver

Super! To list all available endpoints, just run the below command:

$ django-rest-formly list --host 127.0.0.1 --port 8000
Available endpoints:
  tasks: http://127.0.0.1:8000/tasks.json
  users: http://127.0.0.1:8000/users.json

Now let’s have a look into the tasks endpoint, we’ll use for that the form command:

$ django-rest-formly form --host 127.0.0.1 --port 8000 tasks

Now you can use the JSON output, to generate the correspondent form to your endpoint with form validation in place:

/* global angular */
(function() {
  'use strict';

  var app = angular.module('djangoRestFormlyExample', ['formly', 'formlyBootstrap']);

  app.controller('MainCtrl', function MainCtrl(formlyVersion) {
    var vm = this;
    // function assignment
    vm.onSubmit = onSubmit;

    // variable assignment
    vm.author = { // optionally fill in your info below 🙂
      name: 'Wael BEN ZID<span 				data-mce-type="bookmark" 				id="mce_SELREST_start" 				data-mce-style="overflow:hidden;line-height:0" 				style="overflow:hidden;line-height:0" 			></span>'
    };
    vm.exampleTitle = 'Introduction';
    vm.env = {
      angularVersion: angular.version.full,
      formlyVersion: formlyVersion
    };

    vm.model = {
      note: "",
      score: 0
    };
    vm.options = {
      formState: {
        awesomeIsForced: false
      }
    };

    vm.fields = DjangoRestFormly.toFormly(formFieldConfig);

    // function definition
    function onSubmit() {
      alert("You clicked on 'Submit' button");
    }
  });

})();

Hope you enjoyed the tutorial.

Advertisements

Scheduled jobs with Celery, Django and Redis

Setting up a deferred task queue for your Django application can be a pain and it shouldn’t to be. Some “persons” use cron which is not only a bad solution, but this is a disaster. Personally, I use Celery. In this post, I’ll show you how to set-up a deferred task queue for your Django application using Celery.

What’s Celery ?

Celery is an asynchronous task queue/job queue based on distributed message passing. It is focused on real-time operation, but supports scheduling as well.

The promise of Celery is to allow you to run code later, or regularly according to a schedule. Unfortunately, running deferred tasks through Celery is not trivial. But it’s useful and beneficial, as it has a distributed architecture that scales as you need. Any Celery installation is composed of three core components:

  1. Celery client: which used to issue background jobs.
  2. Celery workers: these are the processes responsible to run jobs. Worker can be local or remote, so you can start with a single worker in the same web application server, and later add workers as your traffic and overload grow.
  3. Message broker: The client communicates with the the workers through a message queue, and Celery supports several ways to implement these queues. The most commonly used brokers are RabbitMQ and Redis.

Installing requirements

Fistable, let’s install Redis:

$ sudo apt-get install redis-server

Now, let’s install some python packages:

pip install celery
pip install django-celery

Configuring Django for Celery

Once the installation is completed, you’re ready to set up our scheduler. Let’s configure Celery:

INSTALLED_APPS = (
    'djcelery',
)

BROKER_URL = 'redis://127.0.0.1:6379/0'
BROKER_TRANSPORT = 'redis'
CELERYBEAT_SCHEDULER = 'djcelery.schedulers.DatabaseScheduler'

The above lines is used to configure Celery: which broker you’ll use? Which scheduler for heart beat event ?

As you added djcelery package to your INSTALLED_APPS, you need to create the celery database tables – instructions for that differ depending on your environment, If using South or Migrations (Django >= 1.7) for schema migrations:

$ python manage.py migrate

Otherwise:

$ python manage.py syncdb

Below, the celery.py file that is used for setting up the scheduler for your django project:

# celery.py file
from future import absolute_import

import os
import django

from celery import Celery
from django.conf import settings

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'demo.settings')
django.setup()

app = Celery('Scheduler')

app.config_from_object('django.conf:settings')
app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)

Write some tasks

Let’s assume that you have a task that should be executed periodically, a good example might be a twitter bot or a scraper.

import tweepy

api = tweepy.API()

def get_recent_tweets(query):
    for tweet in tweepy.Cursor(api.search, q=query,
                               rpp=100, result_type="recent",
                               include_entities=True,
                               lang="en").items():
        print tweet.created_at, tweet.text
        # Save tweet into database
        ...

Now, we need to create a Celery task for get_recent_tweets

    ## /project_name/app_name/tasks.py

    from celery.decorators import task

    from utils import twitter

    @task
    def get_recent_tweets(*args):
        # Just an example
        twitter.get_recent_tweets(*args)

N.B: Things can get a lot more complicated than this.

Scheduling it

Now, we have to schedule our tasks. For get_bigdata_tweets task, we will run it every hour, this is an interesting subject that I want to follow, For this purpose, I’ll use celery.beat scheduler. In settings.py file add this code:

from celery.schedules import crontab

CELERYBEAT_SCHEDULER = "djcelery.schedulers.DatabaseScheduler"
CELERYBEAT_SCHEDULE = {
    "get_bigdata_tweets": {
        'task': "bots.twitter.tasks.get_recent_tweets",
        # Every 1 hour
        'schedule': timedelta(seconds=6),
        'args': ("bigdata"),
    },
}
For further details, about scheduler configuration, see documentation.

Merge querysets from different django models

If you were in a situation where you need to merge two querysets from different models into one, you’ve surely see this error:

Cannot combine queries on two different base models.

The solution is to use itertools.chain which make an iterator that is the junction of the given iterators.

from itertools import chain

result_lst = list(chain(queryset1, queryset2))

Now, you can sort the resulting list by any common field, e.g. creation date

from itertools import chain
from operator import attrgetter

result_lst = sorted(
    chain(queryset1, queryset2),
    key=attrgetter('created_at'))

Django ORM: Optimize your code

In the most code that we write, we use Django queries that guarantee all CRUD operations. However, sometimes you will need to retrieve values that are derived by summarizing or aggregating a collection of objects.

Throughout this post, we’ll refer to the following models.

from django.db import models

class URL(models.Model):
    url = models.URLField()

class Tweet(models.Model):
    tweet_id = models.IntegerField()
    text = models.CharField(max_length=140)
    tweeple = models.CharField(max_length=200)
    urls = models.ManyToManyField(URL)

I want to get most popular URLs (most shared URLs between tweets)

def most_popular_url():
    urls = URL.objects.all()
    res = []
    for url in urls:
        res.append({&amp;quot;url&amp;quot;: url.url, &amp;quot;count&amp;quot;: url.tweet_set.count()})
    # Let's sort the result
    result = sorted(res, key=lambda x: -x[&amp;quot;count&amp;quot;])

Let’s test this: We have two test cases:

  1. [TC1] Small database that contains 2000 tweets
  2. [TC2] Medium database that contains 3M tweets

The function takes 8.281s for TC1, and 175m:361s for TC2. This is not acceptable and should be improved. Fortunately, Django resolve this for us by providing a rich ORM API : look at the solution below using the annotate

def most_popular_url_optimized():
    urls = URL.objects.annotate(rank=Count(&amp;quot;tweet&amp;quot;)).order_by(&amp;quot;-rank&amp;quot;)
    res = []
    for url in urls:
        res.append({&amp;quot;url&amp;quot;: url.url, &amp;quot;count&amp;quot;: url.rank})

The second version of most_popular_url takes only 2.361s for TC1 and 3541.5s (59m & 0.25s) for TC2. This example show for us the interest of optimizing database query. As a general advice, try to do every treatment in models or forms instead of views

Django: can’t compare offset-naive and offset-aware datetimes error

I encountered an horrible problem today, and I wants to share the solution with you. So I decided to write this post. The problem was: can’t compare offset-naive and offset-aware datetimes. The origin of this error is a compare between an offset-aware date time, obtained from the database in my case, and an offset-naive date time, this is the default type of python datetime object. Below the line that caused this error:

from datetime import timedelta, datetime

one_hour_ago = datetime.now() - timedelta(hours=1)
if topic.created > one_hour_ago:
    # Do something

So the solution is to make both of compared datetime to either naive or aware datetime object. Fortunately, Django comes with a helper to solve this problem:

>>> from django.utils import timezone
>>> now = timezone.now()
datetime.datetime(2013, 12, 11, 9, 43, 44, 868854, tzinfo=<UTC>)

Now, “now” is an offset-aware datetime object. For non djangonauts developer, you can use pytz python module like that:

>>> from datetime import datetime
>>> import pytz
>>> utc = pytz.UTC
>>> now = utc.localize(datetime.now())
datetime.datetime(2013, 12, 11, 9, 43, 44, 868854, tzinfo=<UTC>)