Introduction

Part 1, Chapter 1

Years ago, I worked for an education technology company. One of our clients, a popular university, wanted us to build them a searchable course catalog, and I took the lead in developing the product. I had written search functionality in the past -- mostly admin pages that performed simple queries -- and I modeled the course search after that. Over the course of several sprints, I built the product and demoed it to the stakeholders. Everyone agreed that it worked the way they expected.

We released the feature and the school deployed the course catalog. Almost immediately the negative feedback started pouring in. "The search is broken. It keeps saying 'No results found'." "I can't find any courses." I began to panic. I was positive that I tested the feature thoroughly. Were the servers down? I logged into the app and confirmed that everything was working. Things reached a boiling point when the client called to complain. "The search is unusable. You need to fix this now."

I decided to check the logs to see what users were searching and what I saw surprised me. The majority of the queries were fragments, abbreviations, and strings of keywords. Users were searching for "intro to math", not "Introduction to Mathematics". I had built the app to find documents that matched the search terms exactly. How could I be so naïve?

I had to re-build the app from scratch in a way that captured what users were actually searching for. In the following weeks, I went down a rabbit hole of search algorithms and technologies. I learned that developing a good search application requires you to be a little bit of a mind reader. You don't match the exact terms the user searches; you gather all the documents that the user probably wants to see and then you rank them by relevance.

In the end, I delivered a much different product that worked a lot better. The client was happy and the users finally found what they were looking for.

Preview

Imagine you have a client who has a sizable wine collection and would like an application that lets them find wines based on input criteria. In this course, we're going to walk through the process of creating that search application from scratch.

The instruction will be given in four parts:

1. Part 1: We'll build a basic Django app that uses simple database queries you may have already encountered.
2. Part 2: We'll dig into more advanced concepts such as full-text search, ranking, and trigrams; and you'll learn how to use them with Django and Postgres.
3. Part 3: We'll develop a simple React client to provide the search controls and display the search results.
4. Part 4: Finally, we'll explore Elasticsearch and see how to use that technology to improve our search while scaling our application.

Our server-side application uses:

• Python (v3.11)
• Django (v4.1)
• Django REST Framework (v3.14)
• Postgres (v15.2)
• Elasticsearch (v7.17)

Client-side:

• React (v18.2)

We'll also use Docker Engine (v20.10) and Docker Compose (v2.15).