Getting Started with Redis and RediSearch
- GSP763
- Overview
- Setup and requirements
- Task 1. Install a Redis database with RediSearch
- Task 2. Connect to Redis and insert data
- Task 3. Query data
- Task 4. Insert, update, delete, and expire documents
- Task 5. Managing indexes
- Task 6. Import sample datasets
- Task 7. Querying the movie dataset
- Task 8. Aggregation
- Task 9. Advanced features
- Congratulations!
This lab was developed with our partner, Redis Labs. Your personal information may be shared with Redis Labs, the lab sponsor, if you have opted-in to receive product updates, announcements, and offers in your Account Profile.
GSP763
Overview
Redis is an open source (BSD licensed), in-memory data structure store, used as a database, cache, and message broker. It is designed for applications requiring the fastest possible response times. To manage data, Redis uses key/pair values, and those keys point to any number of data structures. Some of the more popular data structures include strings, hashes, sets, and lists.
In this lab you will use Redis and RediSearch, an add-on module that enables full-text search and secondary indexing in Redis, to install and run a Redis instance, import example hashes, index them with RediSearch, and query them.
Using Redis hashes
Application developers often use Redis hashes to represent their domain objects. A hash is just a set of field/value pairs. For instance, suppose you wanted to store a catalog of films in your Redis database. You might store the information for a Star Wars film in a hash with the HSET
command:
This command stores a hash at the key "movie:001". The hash contains three fields: title
, director
, and plot
.
Now, if you want to access this data, you need to know the name of the key at which it's stored. To retrieve this hash, you can run the HGETALL command:
This will return all of the movie data for Star Wars.
Using RediSearch
What if you want to retrieve hashes based on their contents, not just based on their key? What if, for example, you want to get a list of all movies directed by George Lucas? RediSearch is the tool that makes this possible.
RediSearch indexes the data you store in your Redis hashes and provides you with a query language for looking up that data. In effect, RediSearch makes Redis into a much more general purpose database by letting you run complex queries and aggregations on your data.
What you'll learn
In this lab, you'll learn how to work with Redis and RediSearch by performing the following tasks:
- Install and run a Redis instance with RediSearch using Docker
- Connect to Redis and insert some sample data
- Index and query the database using
FT.CREATE
andFT.SEARCH
- Insert, update, delete, and expire Redis hashes
- Import movie, theater, and user datasets
- Query the datasets using simple and complex conditions, sorting, pagination, and counting
- Aggregate the datasets using
FT.AGGREGATE
, and apply functions to queries
Setup and requirements
Before you click the Start Lab button
Read these instructions. Labs are timed and you cannot pause them. The timer, which starts when you click Start Lab, shows how long Google Cloud resources will be made available to you.
This hands-on lab lets you do the lab activities yourself in a real cloud environment, not in a simulation or demo environment. It does so by giving you new, temporary credentials that you use to sign in and access Google Cloud for the duration of the lab.
To complete this lab, you need:
- Access to a standard internet browser (Chrome browser recommended).
- Time to complete the lab---remember, once you start, you cannot pause a lab.
How to start your lab and sign in to the Google Cloud console
-
Click the Start Lab button. If you need to pay for the lab, a pop-up opens for you to select your payment method. On the left is the Lab Details panel with the following:
- The Open Google Cloud console button
- Time remaining
- The temporary credentials that you must use for this lab
- Other information, if needed, to step through this lab
-
Click Open Google Cloud console (or right-click and select Open Link in Incognito Window if you are running the Chrome browser).
The lab spins up resources, and then opens another tab that shows the Sign in page.
Tip: Arrange the tabs in separate windows, side-by-side.
Note: If you see the Choose an account dialog, click Use Another Account. -
If necessary, copy the Username below and paste it into the Sign in dialog.
{{{user_0.username | "Username"}}} You can also find the Username in the Lab Details panel.
-
Click Next.
-
Copy the Password below and paste it into the Welcome dialog.
{{{user_0.password | "Password"}}} You can also find the Password in the Lab Details panel.
-
Click Next.
Important: You must use the credentials the lab provides you. Do not use your Google Cloud account credentials. Note: Using your own Google Cloud account for this lab may incur extra charges. -
Click through the subsequent pages:
- Accept the terms and conditions.
- Do not add recovery options or two-factor authentication (because this is a temporary account).
- Do not sign up for free trials.
After a few moments, the Google Cloud console opens in this tab.
Activate Cloud Shell
Cloud Shell is a virtual machine that is loaded with development tools. It offers a persistent 5GB home directory and runs on the Google Cloud. Cloud Shell provides command-line access to your Google Cloud resources.
- Click Activate Cloud Shell at the top of the Google Cloud console.
When you are connected, you are already authenticated, and the project is set to your Project_ID,
gcloud
is the command-line tool for Google Cloud. It comes pre-installed on Cloud Shell and supports tab-completion.
- (Optional) You can list the active account name with this command:
- Click Authorize.
Output:
- (Optional) You can list the project ID with this command:
Output:
gcloud
, in Google Cloud, refer to the gcloud CLI overview guide.
Task 1. Install a Redis database with RediSearch
- To start, you'll need to get a Redis database with RediSearch enabled. For simplicity, this lab will use pre-built Docker images. To start your Redis instance with Docker, in Cloud Shell run the following command:
- In a new cloud shell tab, clone the Getting Started with RediSearch repository (used later for importing datasets in the lab), and install the
redis-cli
, which is a simple program that allows you to send commands to Redis, and read the replies sent by the server, directly from the terminal:
Y
and enter.You now have a Redis instance running with RediSearch installed! In the next sections you will discover the basics.
Task 2. Connect to Redis and insert data
Before creating the index, you will first need to understand the dataset and insert some example entries. For this lab, you will use a simple dataset describing movies. A movie has the following attributes:
Attribute | Description |
---|---|
movie_id |
the unique ID of the movie, internal to this database |
title |
the title of the movie |
plot |
a summary of the movie |
genre |
the genre of the movie (for now a movie will only have single genre) |
release_year |
the year the movie was released (as a numerical value) |
rating |
the average rating from the public (numerical value) |
votes |
the number of votes |
poster |
link to the movie poster |
imdb_id |
ID of the movie in the IMDB database |
As a Redis developer, one of the first things you need to decide is how to structure your data. This starts with key naming.
A common way of naming keys in Redis is to use a specific pattern. For example, when the database includes various business objects like movies, actors, theaters, and users, you can use the following pattern: business_object_type:ID
.
For example:
-
movie:001
would be the key for the movie with the ID001
-
user:001
would be the key for the user with the ID001
For the movies data itself, you should use Redis hashes. A Redis hash will allow the application to structure all of the movie attributes in individual fields; RediSearch will then index those fields based on an index definition.
Insert movies
It's now time to add some data into your database.
- In the new Cloud Shell tab, run the following command to use the
redis-cli
embedded in the container that connects to the server running in the other tab. Make sure to leave the other tab open to keep the Redis instance running.
- You're now ready to insert some data. This example uses movie data stored as Redis hashes, so start by inserting a few movies (note that you can copy these commands rather than manually typing them in):
- Now you can retrieve this data using the movie ID. Run the following command to get the title and rating:
Output:
- You can increment the rating of this movie using the following command:
Output:
But how do you get a movie or list of movies by release year
, rating value
, or title
?
With “core” Redis, you would have to manually index this data using other Redis data structures (e.g., sets). This is relatively complex. With RediSearch, you can simply define an index associated with your data and let the database do all the indexing for you. You can then query your data using the RediSearch query syntax.
RediSearch and indexing
To create an index, you must define a schema specifying the fields to index. For this example, you'll be indexing four fields:
Title
Release Year
Rating
Genre
When creating an index you define:
- Which data you want to index: for instance, all hashes with a key starting with
movies
- Which fields in the hashes you want to index
Create the index
- Create the index with the following command:
Before any running queries, take a closer look at the FT.CREATE
command.
Parameter | Description |
---|---|
idx:movie |
the name of the index |
ON hash |
the type of data structure to be indexed |
PREFIX 1 "movie: |
the prefix of the keys that should be indexed. This is a list, so since you want to only index movie:* keys, the number is 1. With two key prefixes, your definition would look like this: PREFIX 2 "movie:" "tv_show:"
|
SCHEMA ... |
defines the fields, their types, and whether the index should be sortable. As you can see in the command, possible types are TEXT, NUMERIC and TAG. SORTABLE is an additional parameter. |
The RediSearch engine will scan the database using the PREFIX values, and update the index based on the schema definition. This makes it easy to add an index to an existing application that uses hashes. You can find more information about the FT.CREATE command in the documentation.
- Next, look at the index information with the
FT.INFO
command:
You're ready to use the index and query the database.
Test your understanding
Below are multiple choice questions to reinforce your understanding of this lab's concepts. Answer them to the best of your abilities.
Task 3. Query data
In this section, you'll learn how to query data using the FT.SEARCH
command.
- Start by searching for all movies that contain “war”-related information by running the following command:
Output:
The FT.SEARCH
command returns the number of results, then the list of each individual result.
As you can see, the movie Star Wars: Episode V - The Empire Strikes Back is found, even though you used only the word “war” to match “Wars” in the title. This is because the title has been indexed as text, so the field is tokenized and stemmed.
- You can also limit which fields are returned by using the
RETURN
parameter. Run the same query, but this time return only thetitle
andrelease_year
:
Output:
This query does not specify a field, so the word “war” (and related words) is searched in all text fields of the index. You can also limit your query to specific fields.
- To query on a specific field, use the
@field:
syntax:
- Now try a more complex query. Suppose you want all movies containing the string war but NOT the string
jedi
. Adding the string-jedi
(minus) will tell the query engine to exclude fields containing the wordjedi
.
Output:
- Next, query all the movies that contain the string
gdfather
using fuzzy search. You can request fuzzy matching by surrounding your search term with%
signs:
Output:
As you can see, even though the word "godfather" is incorrectly spelled, this query is still possible using a fuzzy matching. Fuzzy matches are performed based on Levenshtein distance (LD).
- Query all
Thriller
movies. Thegenre
field is indexed as a TAG and allows exact match queries. The syntax to query a TAG field is@field_name:{value}
.
Output:
- Next, query all
Thriller
orAction
movies; this is done with the|
symbol:
Output:
To learn more, refer to Tag filters in the Query syntax reference.
- Building on the last query, you can select all
Thriller
orAction
movies that do not haveJedi
in the title:
Output:
Next, query all the movies released between 1970 and 1980 (inclusive). For this, the FT.SEARCH
syntax has two ways to query numeric fields:
- Using the
FILTER
parameter - Using the
@field
in the query string.
- First, try the
FILTER
parameter:
- Now use the
@field
parameter:
Output:
- To exclude a value, prepend it with
(
in the FILTER or query string, for example to exclude 1980:
Summary
- By default, queries apply to all TEXT fields and use the words and their base, stemmed form
- To restrict your query to a specific field, you must use the
@field:
notation - Multiple conditions "intersection" (AND condition) and "union" (OR condition, using
|
) are possible
Test your understanding
Below are multiple choice questions to reinforce your understanding of this lab's concepts. Answer them to the best of your abilities.
Task 4. Insert, update, delete, and expire documents
In this section you will see how RediSearch responds when you insert new documents, update old ones, delete some others, and what happens when documents expire.
As part of this lab you have:
- Created a few movies as Redis hashes (documents) and stored them using the
movie:[ID]
key pattern - Created an index using the
FT.CREATE
command - Queried the data using
FT.SEARCH
When creating the index, using the idx:movie ON hash PREFIX 1 "movie:"
parameter, you are asking the indexing engine to look at all existing keys and index them. As such, new information that matches this pattern/type, will also be indexed.
- Start by counting the number of movies:
Output:
- Now, add a new movie:
- Count them again:
Output:
As you can see, the new movie has been indexed.
- Next, search on any of the indexed fields:
Output:
- Now, run the following to update one of the fields:
- And search for
007
:
Output:
When you delete the hash, the index is also updated, and the same happens if the key is expired (for example, by using a TTL, or Time To Live).
- Set a 20-second TTL on the James Bond movie:
- You can run the following query and see that after 20 seconds, the document does not exist anymore:
The search query will not return any result, showing that the index has been updated:
Output:
Task 5. Managing indexes
In this section, you will learn how to list, inspect, and update your indexes.
- Use the
FT._LIST
command to get a list of all RediSearch indexes of your database:
Output:
- Use the
FT.INFO
command to get information about the index:
Output:
Updating your index definitions
As you are building your application and adding more information to the database, you may need to add new fields to the index. Adding a new field is possible using the FT.ALTER
command.
- Use the
FT.ALTER
command to add a new field to themovie
index:
The WEIGHT
parameter declares the importance of this field when calculating results. This is a multiplication factor (default is 1); so in this example the plot is less important than the title.
- Now, run a query with the new indexed field:
- You can now drop the index using the
FT.DROPINDEX
command:
- Dropping the index does not impact the indexed hashes; this means that the movies are still inside the database. Check this by running the following query:
Output:
FT.DROPINDEX
command.
Test your understanding
Below are multiple choice questions to reinforce your understanding of this lab's concepts. Answer them to the best of your abilities.
Task 6. Import sample datasets
In the previous sections you used just a few movies to perform some basic queries. In this section, you will import existing datasets that include more movies to discover more query features, theaters to discover the geospatial capabilities, and users to do some aggregations.
Movies
The file sample-app/redisearch-docker/dataset/import_actors.redis
is a script that creates 922 Hashes.
The movie hashes contain the following fields:
Field | Description |
---|---|
movie_id |
the unique ID of the movie, internal to this database |
title |
the title of the movie |
plot |
a summary of the movie |
genre |
the genre of the movie (for now a movie will only have single genre) |
release_year |
the year the movie was released (as a numerical value) |
rating |
the average rating from the public (numerical value) |
votes |
the number of votes |
poster |
link to the movie poster |
imdb_id |
ID of the movie in the IMDB database |
Theaters
The file sample-app/redisearch-docker/dataset/import_theaters.redis
is a script that creates 117 Hashes (used for Geospatial queries).
The theater hashes contain the following fields:
Field | Description |
---|---|
theater:id |
the unique ID of the theater, internal to this database (used as the key to the hash) |
name |
the name of the theater |
address |
the street address |
city |
the city (in this sample dataset all the theaters are in New York) |
zip |
the zip code |
phone |
the phone number |
url |
the URL of the theater |
location |
the longitude and latitude used to create the geo-indexed field |
Users
The file sample-app/redisearch-docker/dataset/import_users.redis
is a script that creates 5996 Hashes.
The user hashes contain the following fields:
Field | Description |
---|---|
user:id |
the unique ID of the user |
first_name |
the first name of the user |
last_name |
the last name of the user |
email |
the email address of the user |
gender |
the gender of the user |
country |
the country name |
country_code |
the country code |
city |
the city of the user |
longitude |
the longitude of the user |
latitude |
the latitude of the user |
last_login |
the last login of the user, in EPOCH time |
ip_address |
the IP address of the user |
Import the movies, theaters, and users
- Before importing the data, flush the database using the
FLUSHALL
command:
- The easiest way to import the files is to use
redis-cli
. Open a new Cloud Shell tab and run the following commands:
- Return to the previous Cloud Shell tab, with the
redis-cli
prompt open, and run the following to take a look at the dataset:
Output:
- Get the theater name and location at index 20:
Output:
- Get the data associated with user number
343
:
Output:
You can also use the DBSIZE
command to see how many keys you have in your database.
Create indexes
- Before you can do more queries, you'll need to create the
idx:movie
index with the following command:
- The movies have now been indexed. Run the following command and look at the
num_docs
returned value. The value should be 922.
- Create the
idx:theater
index. This index will mostly be used to show the geospatial capabilities of RediSearch. In the previous examples you created indexes with fields having three different types:
Text
Numeric
Tag
- You will now discover a new type of field:
Geo
. Thetheater
hashes contains a fieldlocation
with the longitude and latitude that will be used in the index as follows:
- The theaters have now been indexed. Run the following command and look at the
num_docs
returned value. The value should be 117.
- Lastly, create the
idx:user
index:
Task 7. Querying the movie dataset
As described earlier in the lab, one of the goals of RediSearch is to provide rich querying capabilities such as simple and complex conditions, sorting, pagination, and counting. This section will provide some examples and insight into the various capabilities of querying a sizable dataset.
Conditions
The best way to start to work with RediSearch's query capabilities is to look at the various conditions options.
- Run the following to find all the movies that contain the word 'heat':
Output:
The first line contains the number of documents (4
) that match the query condition, then the list of movies.
This query is a "fieldless" condition, meaning that the query engine has:
- Searched all of the TEXT fields from the index (
title
andplot
) - For the word
heat
and related words, returnedmovie:736
since it has the wordheated
in the plot. To learn more, refer to the Stemming reference. - Returned the result sorted by the score. Remember that the title has a weight of 1.0, and the plot a weight of 0.5. So when the word or related words are found in the title, the score is bigger.
- Now, find all the movies with a title that contain the word "heat" or related to "heat". This is a field-specific query:
Output:
Sort
A common use case when querying data is to sort the data on a specific field, and paginate over the result.
- Query all the
Action
movies, sorted by release year from newest to the oldest.
Output:
The first line contains the number of documents (186
) that match the query condition.
The FT.SEARCH
command, by default, returns the first ten documents. You will see in the next query how to paginate.
Paginate
- Query all the
Action
movies, sorted by release year from the oldest to the most recent one, returning the record by batch of 100 movies:
Output:
The result is very similar to the previous query:
- 186 documents found
- The first document is the oldest one, released in 1966
- The latest movie of the batch is released in 2014
- To paginate to the next batch, you'll need to change the limit as follows:
Count
- Count the number of
Action
movies. Based on the sample queries you've already seen, if you put theLIMIT 0 0
, RediSearch will return the number of documents that match the query condition:
Output:
Geospatial queries
Suppose you're at MOMA, located at "11 W 53rd St, New York", and you want to find all the theaters located within a 400 meter radius. For this, you need to start with the location (-73.9798156,40.7614367)
(long/lat) of your current location.
- Find theaters, names, and addresses that are at less than 400 meters from MOMA:
Output:
Task 8. Aggregation
A common need for an application, in addition to retrieving information as a document list, is aggregation. For example, if you look at the movie documents, you may want to retrieve the number of movies and group by release year, starting with the most recents ones. For this, RediSearch provides the FT.AGGREGATE
command.
GROUPBY & SORTBY
- Start by aggregating the number of movies by year:
Output:
- Building on the last command, aggregate the number of movies by year, from the newest to the oldest:
Output:
Apply functions
The idx:user
index contains the last_login
field. This field stores the last login time as an EPOCH timestamp. RediSearch aggregation allows you to apply transformations to each record. This is done using the APPLY parameter.
For this example, you can use a date/time function to extract the month and year from the timestamp.
- Run the following command to retrieve the number of logins per year and month:
Output:
- Use filtering to retrieve the number of logins per month for the year 2020. This is similar to the previous query with the addition of a filter on the year:
Output:
Test your understanding
Below are multiple choice questions to reinforce your understanding of this lab's concepts. Answer them to the best of your abilities.
Task 9. Advanced features
In the previous examples, the indices have been created using a PREFIX
, where all the keys matching the data structure type and prefix are indexed.
But you can also create an index using a filter. For example, you can create an index with all the "Drama" movies released between 1990 and 2000 (2000 not included).
The FILTER
expression uses the aggregation filter syntax. Here's a filter expression on genre and release year:
- Here's how to create a drama movie index using this
FILTER
expression:
- Run the
FT.INFO idx:drama
command to look at the index definitions and statistics:
Notes:
- The
PREFIX
is not optional. - In this application, this index is not useful since you can get the same data from the
idx:movie
Finally, you can check that the index has been correctly initialized by running the following queries that should return the same number of documents.
- Check the number of documents for
idx:drama
:
Output:
- Now check the number of documents for
idx:movie
:
Output:
Congratulations!
You've learned a bit more about Redis and the powerful indexing capabilities of RediSearch. To review, you should now know how to install and run a Redis instance with RediSearch using Docker. You should also know how to import a few example hashes, index them with RediSearch, and query them. Finally, you should understand some of RediSearch's aggregation features.
Next steps / Learn more
- Redis Labs on the Google Cloud Marketplace!
Google Cloud training and certification
...helps you make the most of Google Cloud technologies. Our classes include technical skills and best practices to help you get up to speed quickly and continue your learning journey. We offer fundamental to advanced level training, with on-demand, live, and virtual options to suit your busy schedule. Certifications help you validate and prove your skill and expertise in Google Cloud technologies.
Manual Last Updated December 27, 2023
Lab Last Tested October 11, 2023
Copyright 2024 Google LLC All rights reserved. Google and the Google logo are trademarks of Google LLC. All other company and product names may be trademarks of the respective companies with which they are associated.