Skip to content

Multi-field Querying#

2024 iThome Ironman Contest

This is the 27th article in the Django Ninja tutorial series.

In the previous article, we learned about Django ORM's Q object and Django Ninja's FilterSchema, but it felt like we only learned half of the latter.

Most discussions were about the parameter definition method using FilterSchema in the view function—which is indeed important, but it is only a part of FilterSchema.

This article will complete the remaining content:

  1. Perfecting FilterSchema: Using a "more idiomatic" syntax to release the true power of FilterSchema.
  2. Implementing more advanced field querying: Multi-field querying - filtering date ranges.
  3. Additionally implementing the "cross-field validation" learned in the 20th article: Validating whether the date range of the query parameters is valid.

It looks like another article packed with information. Without further ado, let's get started!

For all code changes in this article, please refer to this PR.

GitHub Example Project#

👉 Django-Ninja-Tutorial

1. Migrating Query Logic to FilterSchema#

Do you remember the code implementation from our previous article?

Even though we defined a FilterSchema, the code in the view function not only didn't decrease, but actually increased! (Although query logic also increased because we query two fields simultaneously)

@router.get(...)
@paginate(CustomPagination)
def get_posts(
    request: HttpRequest,
    filters: PostFilterSchema = Query(),  # Use FilterSchema
) -> QuerySet[Post]:
    """
    Get post list
    """
    posts = Post.objects.all()
    if filters.query:
        q = Q(title__icontains=filters.query) | \
            Q(content__icontains=filters.query)
        posts = posts.filter(q)
    return posts

This is simply absurd 🐸

That is because that's not how FilterSchema is meant to be used!

The "Correct" Usage of FilterSchema#

We should package query logic into the FilterSchema as much as possible. This makes the view function cleaner and achieves the effect of "separation of concerns".

Let's look at a more reasonable implementation—migrating query logic to FilterSchema:

class PostFilterSchema(FilterSchema):
    query: str | None = Field(
        None,
        q=['title__icontains', 'author__username__icontains'],
        min_length=2,
        max_length=10,
    )

The major change is the Field part of the query field, which now includes the q= parameter content:

q=["title__icontains", "author__name__icontains"]

Looks familiar, right? Yes, they are actually condition expressions of the Q object. Django Ninja will automatically invoke Q objects behind the scenes to execute these queries.

Reminder From Mypy#

Once you use the q= parameter, Mypy will remind you:

Unexpected keyword argument "q" for "Field"

It is not wrong, because Pydantic Field indeed does not have this parameter—it is implemented by Django Ninja itself.

You can ignore it or add the necessary annotations.

View Function Simplification#

As a result, the view function only needs to be written like this:

...
def get_posts(
    request: HttpRequest,
    filters: PostFilterSchema = Query(),
) -> QuerySet[Post]:
    """
    Get post list
    """
    posts = Post.objects.select_related('author')
    posts = filters.filter(posts)
    return posts

Isn't that much simpler?

Because the query logic is "separated" from the view function, the view function's responsibility is single and easier to maintain.

2. Multi-Field Querying: Adding Date Filtering#

New requirement: Besides querying the post title or author name, we now want to add filtering on the "post date"!

We will introduce two new URL query parameters:

  • start_date
  • end_date

Both will be used to query and filter the created_at field (i.e., post date) in the Post model, to filter post data within a specific time range.

There is also an additional requirement: they must be "all or nothing"—either neither is filled, or both must be. Since this is a date range query, there must be a start and an end.

This is also a classic "multi-field" query.

Adding Code#

Here is the FilterSchema with the above logic added:

class PostFilterSchema(FilterSchema):
    query: str | None = Field(
        None, q=["title__icontains", "author__username__icontains"])
    start_date: str | None = Field(None, q="created_at__gte")
    end_date: str | None = Field(None, q="created_at__lte")

Here, both start_date and end_date are query conditions for the model field created_at.

So we use created_at__gte and created_at__lte to describe the filtering logic (each maps to its respective Q object) to filter data that meets the conditions.

What about the view function? You guessed it right—no changes needed at all!

This is the beauty of using FilterSchema.

API Documentation Rendering Issues#

Interestingly, when I tried to add documentation examples to these query parameters like this:

start_date: str | None = Field(
    None, q='created_at__gte', examples=['2021-01-01']
)

Viewing the API documentation returns:

😱 Could not render Parameters, see the console.

But writing example='2021-01-01' works fine.

This might be a bug in the integration between Django Ninja and Pydantic. Let's skip it for now!

Client Query Examples#

When we want to query posts within a certain period of time, we can use the following URL query parameters:

?start_date=2023-01-01&end_date=2023-01-31

This makes it easy to query all posts in January 2023.

By the way, because the time in the date is not specified, it defaults to 00:00:00. Therefore, if you fill in the same day, you won't query anything.

This is a detail that needs to be improved or re-adjusted. A common practice is to add 1 day to the end_date inside the program, but I chose to forbid the two from being the same directly XD. What you actually do depends on your requirements.

Besides simple period queries, we can also query posts of a certain author within a certain period of time. Taking author Alice as an example:

?start_date=2023-01-01&end_date=2023-01-31&query=alice

The result will display all posts of Alice in January 2023.

Default Query Condition Relationships in FilterSchema#

This part must be introduced specifically. According to the documentation, by default:

  • Field-level expressions are joined together using OR operator.
  • The fields themselves are joined together using AND operator.

This means that multiple Q expressions within a single field are in an OR relationship, such as query above:

q=['title__icontains', 'author__username__icontains']

This queries post title or author name.

On the other hand, conditions in different fields (if both exist) are in an AND relationship—both must be met. Therefore, the author name and date range conditions must both be satisfied.

These default logics can be changed. For details, please refer to the documentation mentioned above.

3. Date Range Validation#

In this example, in addition to field queries, we also want to ensure that the start date input by the user must be earlier than the end date.

Furthermore, the query values of both fields must be "all or nothing" (no validation is required if neither is provided).

This requirement looks very familiar—isn't it the "cross-field validation" mentioned in the 20th article?

Yes, we will implement this using Pydantic's model_validator, which allows us to perform custom logical checks on input data during the validation process.

Implementing Date Range Validation Using Pydantic model_validator#

The code is a bit long, so let's look straight at the key points:

class PostFilterSchema(FilterSchema):
    ...
    start_date: str | None = Field(None, q='created_at__gte')
    end_date: str | None = Field(None, q='created_at__lte')

    @model_validator(mode='after')
    def check_date_range(self) -> Self:
        # If both start_date and end_date are None, do not perform any checks
        if self.start_date is None and self.end_date is None:
            return self

        if not all([self.start_date, self.end_date]):
            raise ValueError('Start and end dates must either both be provided or both be omitted')

        try:
            start_date_dt = datetime.strptime(self.start_date, '%Y-%m-%d')
            end_date_dt = datetime.strptime(self.end_date, '%Y-%m-%d')
        except ValueError:
            raise ValueError('Invalid date format; expected YYYY-MM-DD')

        if start_date_dt > end_date_dt:
            raise ValueError('Start date must be earlier than end date')

        return self

For query conditions, we use Pydantic's model_validator to perform cross-field validation, ensuring the dates entered by the user are valid and reasonable.

In fact, cross-field validation often requires considering many details; otherwise, it might leave loopholes and indirectly create new bugs.

This example is a classic case.

We must comprehensively consider various possible input scenarios, including whether the date format is correct, whether the date range is reasonable, and whether both date fields exist or are both empty.

Detailed validation logic enhances the reliability of the API, avoiding unexpected behaviors in the system due to invalid or unreasonable inputs, whereas coarse logic does the opposite.

Testing Error Responses#

PS: The example code still used ValueError to throw errors; I did not change it to Django's ValidationError. (This has been fixed in the latest version)

However, the following responses simulate Django's ValidationError to reduce unnecessary repetition.

  1. Entering start date only: (This has a low probability of happening, as the frontend usually restricts it)
{
    "detail": "Start and end dates must either both be provided or both be omitted"
}
  1. Entering an invalid date, such as 2023-02-30:
{
    "detail": "Invalid date format; expected YYYY-MM-DD"
}
  1. Entering an invalid date range, such as start_date=2023-01-31&end_date=2023-01-01:
{
    "detail": "Start date must be earlier than end date"
}

Summary and Next Steps#

In these two articles, we introduced the effective use of FilterSchema, completed multi-field queries and date filtering, and demonstrated how to use model_validator to strengthen data validation, ensuring the correctness of query logic.

We also examined code examples for these scenarios to clarify the purpose and effect of each step.

In the next chapter, we will explore the authentication mechanism in Django Ninja and introduce how to conduct unit testing with pytest, both of which are indispensable elements in backend development.