Requests and Routing

Chapter 2


In the second chapter of the course, we will build the most important parts of the framework:

  1. The request handlers (think Django views)
  2. Routing -- both simple (like /books/) and parameterized (like /books/{id}/)

Now, before I start doing something new, I like to think about the end result. In this case, at the end of the day, we want to be able to use this framework in production and thus we want our framework to be served by a fast, lightweight, production-level application server. I have been using Gunicorn in all of my projects the past few years, and I am very satisfied with the results. So, we'll go with Gunicorn as well.

Gunicorn is a WSGI HTTP Server, so it expects a specific entrypoint to our application, just like you learned in the last chapter.

To recap, to be WSGI-compatible, you need a callable object (a function or a class) that expects two parameters (environ and start_response) and returns an iterable response.

With those details out of the way, let's get started with our awesome framework.

Requests

First, think of a name for your framework and create a folder with that name.

We'll be using the name bumbo throughout this course. You can use that same name or come up with something new.

$ mkdir bumbo

Change directories to that folder. Create a virtual environment and activate it:

$ cd bumbo
$ python3.7 -m venv venv
$ source venv/bin/activate

Now, create a file named app.py where we will store our entrypoint for Gunicorn:

(venv)$ touch app.py

Inside app.py, add the following WSGI-compatible dummy function to see if it works with Gunicorn:

# app.py


def app(environ, start_response):
    response_body = b"Hello, World!"
    status = "200 OK"
    start_response(status, headers=[])
    return iter([response_body])

It's nearly the same as the function you created in the previous chapter but much simpler. Instead of listing all the environment variables, it simply says "Hello, World!".

Now, to test, first install Gunicorn:

(venv)$ pip install gunicorn

Then, run the code:

(venv)$ gunicorn app:app

In the above command, the first app (left of the colon) is the file which you created and the second app (right of the colon) is the name of the function you just wrote. If all is good, you will see the following output in your console:

[2019-10-09 02:38:18 -0600] [24662] [INFO] Starting gunicorn 19.9.0
[2019-10-09 02:38:18 -0600] [24662] [INFO] Listening at: http://127.0.0.1:8000 (24662)
[2019-10-09 02:38:18 -0600] [24662] [INFO] Using worker: sync
[2019-10-09 02:38:18 -0600] [24665] [INFO] Booting worker with pid: 24665

If you see this, open your browser and go to http://localhost:8000. You should see our good old friend: the Hello, World! message. Awesome! You'll build the rest of the framework off of this function.

Now, you will need quite a few helper methods, so let's turn this function into a class since they are much easier to write inside a class:

Create an api.py file:

(venv)$ touch api.py

Inside this file, create the following API class:

# api.py


class API:
    def __call__(self, environ, start_response):
        response_body = b"Hello, World!"
        status = "200 OK"
        start_response(status, headers=[])
        return iter([response_body])

We'll look at what this class does together in a bit.

But first, delete everything inside app.py and write the following:

# app.py

from api import API


app = API()

Restart (or re-run) Gunicorn and check the result in the browser. It should be the same as before since we converted our function named app to a class called API and overrode its __call__ method.

It's worth noting that the __call__ method is called when you call the instances of this class:

app = API()
app()   #  this is where __call__ could be called

So, why are we only creating an instance of the API but not calling it in app.py? Simple: The calling of the instance of the API class is the responsibility of the web server (e.g., Gunicorn). In other words, it will be called when you run gunicorn app:app.

Now that you created the class, let's make the code look a bit more elegant because all those bytes (i.e., b"Hello World") and the start_response function seem a bit confusing, right?

Thankfully, there is a cool package called WebOb that provides classes for HTTP requests and responses by wrapping the WSGI request environment and response status, headers, and body. By using this package, we can pass the environ and start_response to the classes, which are provided by this package, and not have to deal with them ourselves.

Before we continue, take a few minutes to look over the WebOb documentation. Make sure you understand how the Request and Response classes work. Briefly review the rest of the API as well.

Let's refactor this code!

Refactoring

First, install WebOb:

(venv)$ pip install webob

Import the Request and Response classes at the beginning of the api.py file:

# api.py

from webob import Request, Response

...

And now you can use them inside the __call__ method:

# api.py

from webob import Request, Response


class API:
    def __call__(self, environ, start_response):
        request = Request(environ)

        response = Response()
        response.text = "Hello, World!"

        return response(environ, start_response)

Looks much better!

Restart Gunicorn and you should see the same result as before. The best part here is that the code above requires little explanation since it's mostly self-explanatory. You created a request and a response and then returned that response. Awesome!

Did you notice that we are not yet using the request? Let's change that. We can use it to get the user agent info from it. At the same time, we can refactor the response creation into its own method called handle_request. You will see why this is better later.

# api.py

from webob import Request, Response


class API:
    def __call__(self, environ, start_response):
        request = Request(environ)

        response = self.handle_request(request)

        return response(environ, start_response)

    def handle_request(self, request):
        user_agent = request.environ.get("HTTP_USER_AGENT", "No User Agent Found")

        response = Response()
        response.text = f"Hello, my friend with this user agent: {user_agent}"

        return response

Restart Gunicorn and you should see the new message in the browser.

For example:

Hello, my friend with this user agent:
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:67.0) Gecko/20100101 Firefox/67.0

Did you see it? Cool. Let's move on.

Routing

At this point, your framework handles all the requests in the same way: Whatever request it receives, it simply returns the same response which is created in the handle_request method. Ultimately, it needs to be dynamic though. That is, it needs to serve the request coming from /home/ differently than the one coming from /about/.

Simple

To that end, inside app.py, create two methods that will handle those two requests:

# app.py

from api import API


app = API()


def home(request, response):
    response.text = "Hello from the HOME page"


def about(request, response):
    response.text = "Hello from the ABOUT page"

Now, you need to somehow associate the two methods with the above mentioned paths -- /home/ and /about/. I like the Flask way of managing this with decorators, which looks something like this:

# app.py

from api import API


app = API()


@app.route("/home")
def home(request, response):
    response.text = "Hello from the HOME page"


@app.route("/about")
def about(request, response):
    response.text = "Hello from the ABOUT page"

Make the above changes to app.py.

So, what do you think? Look good? Let's create and implement the decorators!

As you can see, the route method above is a decorator that accepts a path and wraps the methods. Implement it in the API class like so:

# api.py

from webob import Request, Response


class API:
    def __init__(self):
        self.routes = {}

    def __call__(self, environ, start_response):
        ...

    def route(self, path):
        def wrapper(handler):
            self.routes[path] = handler
            return handler

        return wrapper

    def handle_request(self, request):
        ...

What's happening?

First, In the __init__ method, you defined a dict called self.routes where the framework will store paths as keys and handlers as values.

That dict will look something like this:

{
    "/home": <function home at 0x1100a70c8>,
    "/about": <function about at 0x1101a80c3>
}

Then, in the route method, you took a path as an argument and in the wrapper method you added this path in the self.routes dictionary as a key and the handler as a value.

At this point, you have all the pieces of the puzzle. You have the handlers and the paths associated with them. Now, when a request comes in, you need to check its path, find an appropriate handler, call that handler, and return an appropriate response.

Refactor the handle_request method to do just that:

# api.py

from webob import Request, Response


class API:
    ...

    def handle_request(self, request):
        response = Response()

        for path, handler in self.routes.items():
            if path == request.path:
                handler(request, response)
                return response

Here, you iterated over self.routes and compared paths with the path of the request. If there is a match, you then called the handler associated with that path.

You should now have:

# api.py

from webob import Request, Response


class API:
    def __init__(self):
        self.routes = {}

    def __call__(self, environ, start_response):
        request = Request(environ)

        response = self.handle_request(request)

        return response(environ, start_response)

    def route(self, path):
        def wrapper(handler):
            self.routes[path] = handler
            return handler

        return wrapper

    def handle_request(self, request):
        response = Response()

        for path, handler in self.routes.items():
            if path == request.path:
                handler(request, response)
                return response

Restart Gunicorn again and try those paths in the browser:

  1. http://localhost:8000/home
  2. http://localhost:8000/about

You should see the corresponding messages. Pretty cool, right?

However, what happens if the path is not found?

Try http://localhost:8000/nonexistent/. You should see a big, ugly error message: Internal Server Error. If you turn to your console, you should see the issue:

TypeError: 'NoneType' object is not callable

You should see the same TypeError for the request for the favicon:

[2019-10-09 03:37:19 -0600] [28562] [ERROR] Error handling request /favicon.ico

To solve this, create a method that returns a simple HTTP response of "Not found.", with a 404 status code:

# api.py

from webob import Request, Response


class API:
    ...

    def default_response(self, response):
        response.status_code = 404
        response.text = "Not found."

    ...

Now, use it in the handle_request method:

# api.py

from webob import Request, Response


class API:
    ...

    def handle_request(self, request):
        response = Response()

        for path, handler in self.routes.items():
            if path == request.path:
                handler(request, response)
                return response

        self.default_response(response)
        return response

Restart Gunicorn and try some nonexistent routes. You should see a lovely "Not found." page instead of the previous, unhandled error.

Next, refactor out finding a handler to its own method for the sake of readability:

# api.py

from webob import Request, Response


class API:
    ...

    def find_handler(self, request_path):
        for path, handler in self.routes.items():
            if path == request_path:
                return handler

    ...

Just like before, it iterates over self.route, comparing paths with the request path, and returns the handler if the paths are the same or None if no handler is found.

Now, you can use it in your handle_request method:

# api.py

from webob import Request, Response


class API:
    ...

    def handle_request(self, request):
        response = Response()

        handler = self.find_handler(request_path=request.path)

        if handler is not None:
            handler(request, response)
        else:
            self.default_response(response)

        return response

Much better, right? Fairly self-explanatory too.

Restart Gunicorn to ensure that everything works just like before.

Parameterized

At this point, you have routes and handlers. It works, but the routes are simple. They don't support keyword parameters in the URL path. What if we wanted to have a route of @app.route("/hello/{person_name}") and be able to use person_name inside the handlers?

For example:

def say_hello(request, response, person_name):
    resp.text = f"Hello, {person_name}"

For that, if someone goes to the /hello/Matthew/, the framework needs to be able to match this path with the registered /hello/{person_name}/ and find the appropriate handler. Thankfully, there is a package called Parse that does exactly that for you. Go ahead and install it:

(venv)$ pip install parse

Test this package out in the Python interpreter. First, open the Python interpreter by running python within your virtual environment, and then test out Parse like so:

>>> from parse import parse
>>> result = parse("Hello, {name}", "Hello, Matthew")
>>> print(result.named)
{'name': 'Matthew'}

As you can see, it parsed the string Hello, Matthew and was able to identify that Matthew corresponds to the provided {name}.

Now, use it in your find_handler method to find not only the method that corresponds to the path but also the keyword params that were provided. Make sure that you import it first:

# api.py

from parse import parse
from webob import Request, Response


class API:
    ...

    def find_handler(self, request_path):
        for path, handler in self.routes.items():
            parse_result = parse(path, request_path)
            if parse_result is not None:
                return handler, parse_result.named

        return None, None

    ...

The find_handler method still iterates over self.routes; but, now instead of comparing the path to the request path, it tries to parse it and if there is a result, it returns both the handler and keyword params as a dictionary.

Now, you can use this inside handle_request to send params to the handlers:

# api.py

from parse import parse
from webob import Request, Response


class API:
    ...

    def handle_request(self, request):
        response = Response()

        handler, kwargs = self.find_handler(request_path=request.path)

        if handler is not None:
            handler(request, response, **kwargs)
        else:
            self.default_response(response)

        return response

So, you are now getting both handler and kwargs from self.find_handler and passing kwargs to the handler via **kwargs.

You should now have:

# api.py

from parse import parse
from webob import Request, Response


class API:
    def __init__(self):
        self.routes = {}

    def __call__(self, environ, start_response):
        request = Request(environ)

        response = self.handle_request(request)

        return response(environ, start_response)

    def route(self, path):
        def wrapper(handler):
            self.routes[path] = handler
            return handler

        return wrapper

    def default_response(self, response):
        response.status_code = 404
        response.text = "Not found."

    def find_handler(self, request_path):
        for path, handler in self.routes.items():
            parse_result = parse(path, request_path)
            if parse_result is not None:
                return handler, parse_result.named

        return None, None

    def handle_request(self, request):
        response = Response()

        handler, kwargs = self.find_handler(request_path=request.path)

        if handler is not None:
            handler(request, response, **kwargs)
        else:
            self.default_response(response)

        return response

Write a handler with this type of route and try it out in app.py:

# app.py

from api import API


app = API()

...

@app.route("/hello/{name}")
def greeting(request, response, name):
    response.text = f"Hello, {name}"

Full file:

# app.py

from api import API


app = API()


@app.route("/home")
def home(request, response):
    response.text = "Hello from the HOME page"


@app.route("/about")
def about(request, response):
    response.text = "Hello from the ABOUT page"

@app.route("/hello/{name}")
def greeting(request, response, name):
    response.text = f"Hello, {name}"

Restart Gunicorn and navigate to http://localhost:8000/hello/Matthew. You should see the wonderful message of Hello, Matthew. Try http://localhost:8000/hello/Mike and you should see Hello, Mike. Awesome, right?

Add a couple more such handlers on your own.

You can also indicate the type of the given params.

For example, you can define a digit type to an age param:

@app.route("/tell/{age:d}")

Another example:

@app.route("/sum/{num_1:d}/{num_2:d}")
def sum(request, response, num_1, num_2):
    total = int(num_1) + int(num_2)
    response.text = f"{num_1} + {num_2} = {total}"

What happens if you pass in a non-digit? It will not be able to parse it and our default_response will do its job. Try this out.

Conclusion

In this chapter, you started writing your own framework and built the most important features:

  1. Request handlers
  2. Simple and parameterized routing

You also learned about and how to use the WebOb and the third-party package named Parse.

We'll look at handling duplicate routes in the next chapter. See you there!

Have you figured out why we refactored out the response creation into its own method called handle_request yet?




Mark as Completed