EXPLAIN EXTENDED http://explainextended.com How to create fast database queries Tue, 09 Mar 2010 18:43:36 +0000 http://wordpress.org/?v=2.9.2 en hourly 1 Aggregates and LEFT JOIN http://explainextended.com/2010/03/05/aggregates-and-left-join/ http://explainextended.com/2010/03/05/aggregates-and-left-join/#comments Fri, 05 Mar 2010 20:00:09 +0000 Quassnoi http://explainextended.com/?p=4548 From Stack Overflow:

I have a table product with products and table sale with all sale operations that were done on these products.

I would like to get 10 most often sold products today and what I did is this:

SELECT  p.*, COUNT(s.id) AS sumsell
FROM    product p
LEFT JOIN
        sale s
ON      s.product_id = p.id
        AND s.dt >= '2010-01-01'
        AND s.dt < '2010-01-02'
GROUP BY
        p.id
ORDER BY
        sumsell DESC
LIMIT 10

, but performance of it is very slow.

What can I do to increase performance of this particular query?

The query involves a LEFT JOIN which in MySQL world means that products will be made leading in the query. Each record of product will be taken and checked against sale table to find out the number of matching records. If no matching records are found, 0 is returned.

Let’s create the sample tables:

Table creation details

The table contains 500,000 products and 500,000 random sales (1,440 sales per day).

Now, let’s run the query similar to the author’s one. I adjusted the period so that fewer than 10 actual sales were made during the period and LEFT JOIN records can be seen in the table:

SELECT  p.*, COUNT(s.id) AS sumsell
FROM    product p
LEFT JOIN
        sale s
ON      s.product_id = p.id
        AND s.dt >= '2010-01-01'
        AND s.dt < '2010-01-01 00:07:00'
GROUP BY
        p.id
ORDER BY
        sumsell DESC, p.id
LIMIT 10

View query details

The query runs for 3 seconds.

We see that, first, product is made leading, and, second, only a part of the index on sale (product, dt) is used: each sale is only filtered on product, not on date.

Since there were only 7 sales during the period we have chosen, it would be a wise decision to make sale leading in the join so that it could be filtered on date and the resulting recordset was then joined to product. This would result in at most 7 PRIMARY KEY seeks instead of 500,000 range scans and would be much more efficient.

However, this is only possible with the INNER JOIN, and if there are less then 10 products sold within the time period, we will not see the rest.

To work around this, we need to emulate the LEFT JOIN:

  1. Find the products sold within the time period, using an INNER JOIN of product with the resultset containg aggregated sales.

  2. Find the products not sold within the time period, using NOT EXISTS predicate.

  3. Concatenate the two resultsets using UNION ALL.

The step 2 implies that product is leading again, so normally it would not be much of improvement. But in our case, we don’t need the whole recordset, we only need the top 10 sales.

So we can just order and limit the recordsets retrieved on steps 1 and 2 to ten records each, concatenate them, then order and limit the resulting recordset again to ten records.

The second resultset will contain a hardcoded 0 in the sumsell, so we just need to order it on product.id. Since product is an InnoDB table and product.id is a clustered PRIMARY KEY, this is not a problem.

Here’s the query:

SELECT  p.*, sumsell
FROM    (
        SELECT  *
        FROM    (
                SELECT  product_id, sumsell
                FROM    (
                        SELECT  product_id, COUNT(*) AS sumsell
                        FROM    sale si
                        WHERE   dt >= '2010-01-01'
                                AND dt < '2010-01-01 00:07:00'
                        GROUP BY
                                product_id
                        ) si
                ORDER BY
                        sumsell DESC, product_id
                LIMIT 10
                ) q1
        UNION ALL
        SELECT  *
        FROM    (
                SELECT  p.id, 0
                FROM    product p
                WHERE   NOT EXISTS
                        (
                        SELECT  NULL
                        FROM    sale si
                        WHERE   product_id = p.id
                                AND dt >= '2010-01-01'
                                AND dt < '2010-01-01 00:07:00'
                        )
                ORDER BY
                        p.id
                LIMIT 10
                ) q2
        ORDER BY
                sumsell DESC, product_id
        LIMIT 10
        ) q
JOIN    product p
ON      p.id = q.product_id

View query details

This query completes in less than 7 ms (which is comparable to the time measurement error).

]]> http://explainextended.com/2010/03/05/aggregates-and-left-join/feed/ 0 SQL Server: deleting with self-referential FOREIGN KEY http://explainextended.com/2010/03/03/sql-server-deleting-with-self-referential-foreign-key/ http://explainextended.com/2010/03/03/sql-server-deleting-with-self-referential-foreign-key/#comments Wed, 03 Mar 2010 20:00:45 +0000 Quassnoi http://explainextended.com/?p=4539 From Stack Overflow:

I have an SQL Server table defined as below:

TestComposite
id (PK) siteUrl (PK) name parentId
1 site1 Item1 NULL
2 site1 Item2 NULL
3 site1 Folder1 NULL
4 site1 Folder1.Item1 3
5 site1 Folder1.Item2 3
6 site1 Folder1.Folder1 3
7 site1 Folder1.Folder1.Item1 6

Items and folders are stored inside the same table

If an item is inside a folder, the parentID column is the id of the folder.

I would like to be able to DELETE CASCADE items/folders when I delete a folder.

I tried to define a constraint similar to:

ALTER TABLE [TestComposite]
ADD CONSTRAINT fk_parentid
FOREIGN KEY (ParentID, SiteUrl)
REFERENCES [TestComposite] (ID, SiteUrl) ON DELETE CASCADE

, but it gives me this error:

Introducing FOREIGN KEY constraint 'fk_parentid' on table 'TestComposite' may cause cycles or multiple cascade paths. Specify ON DELETE NO ACTION or ON UPDATE NO ACTION, or modify other FOREIGN KEY constraints.

SQL Server does support chained CASCADE updates, but does not allow one table to participate more that once in a chain (i. e. does not allow loops).

SQL Server, unlike most other engines, optimizes cascading DML operations to be set-based which requires building a cycle-free DML order (which you can observe in the execution plan). With the loops, that would not be possible.

However, it is possible to define such a constraint without cascading operations, and with a little effort it is possible to delete a whole tree branch at once.

Let’s create a sample table:

Table creation details

This table contains 50,000 records forming a hierarchy.

If we try to delete an entry that has some children, we’ll fail:

DELETE
FROM    [20100303_cascade].TestComposite
WHERE   id = 42
        AND siteUrl = 'site1'

Msg 547, Level 16, State 0, Line 1
The DELETE statement conflicted with the SAME TABLE REFERENCE constraint "fk_TestComposite_self". The conflict occurred in database "test", table "20100303_cascade.TestComposite".
The statement has been terminated.

To delete an item and all of its children, we should build a hierarchical query to retrieve the whole branch, and delete the branch all at once.

To do it, we just semi-join the table to the results of the recursive CTE:

WITH    q AS
        (
        SELECT  id, siteUrl
        FROM    [20100303_cascade].TestComposite
        WHERE   id = 42
                AND siteUrl = 'site1'
        UNION ALL
        SELECT  tc.id, tc.siteUrl
        FROM    q
        JOIN    [20100303_cascade].TestComposite tc
        ON      tc.parentID = q.id
                AND tc.siteUrl = q.siteUrl
        )
DELETE
FROM    [20100303_cascade].TestComposite
OUTPUT  DELETED.*
WHERE   EXISTS
        (
        SELECT  id, siteUrl
        INTERSECT
        SELECT  id, siteUrl
        FROM    q
        )

View query results

We see that the whole branch of 31 records was deleted all at once, without violating the FOREIGN KEY.

Unlike some other systems, we don’t have to worry about the order the records are deleted, since all SQL Server referential constraints are deferred till the end of the query.

]]> http://explainextended.com/2010/03/03/sql-server-deleting-with-self-referential-foreign-key/feed/ 0 PostgreSQL: using recursive functions in nested sets http://explainextended.com/2010/03/02/postgresql-using-recursive-functions-in-nested-sets/ http://explainextended.com/2010/03/02/postgresql-using-recursive-functions-in-nested-sets/#comments Tue, 02 Mar 2010 20:00:45 +0000 Quassnoi http://explainextended.com/?p=4528 In the previous article, I discussed a way to improve nested sets model in PostgreSQL.

The approach shown in the article used an analytical function to filter all immediate children of a node in a recursive CTE.

This allowed us to filter a node’s children on the level more efficiently than R-Tree or B-Tree approaches do (since they rely on COUNT(*)).

That solution was pure SQL and it was quite fast, but not optimal.

The drawback of that solution is that it still needs to fetch all children of a node to apply the analytic function to them. This can take much time for the top of the hierarchy. And since the top of the hierarchy is what is what usually shown at the start page, it would be very nice to improve this query yet a little more.

We can do it by creating and using a simple recursive SQL function. This function does not even require PL/pgSQL to be enabled.

Let’s create a sample table:

Table creation details

If we run the query introduced in the previous article to fetch all children up to level 2 from a really top node, we get the following results:

WITH    RECURSIVE
        q AS
        (
        SELECT  id, lft, rgt, 1 AS lvl
        FROM    t_hierarchy
        WHERE   id = 1
        UNION ALL
        SELECT  *
        FROM    (
                SELECT  DISTINCT ON (MAX(hc.rgt) OVER (PARTITION BY q.id ORDER BY hc.lft)) hc.id, hc.lft, hc.rgt, lvl + 1
                FROM    q
                JOIN    t_hierarchy hc
                ON      hc.lft > q.lft
                        AND hc.lft < q.rgt
                WHERE   lvl <= 2
                ORDER BY
                        MAX(hc.rgt) OVER (PARTITION BY q.id ORDER BY hc.lft), hc.lft
                ) q2
        )
SELECT  *
FROM    q

View query details

This runs for almost 15 seconds: too much.

This can be improved by exploiting these two properties of the nested sets model:

  1. The first immediate child of a node is the node holding the first lft next to the node’s lft

  2. The next sibling of a node is the node holding the first lft next to the node’s rgt

If we recursively traverse through the nodes, we can find the first child as well as all of its siblings. This is enough to build a hierarchy, and level filter can be implemented merely by limiting the recursion depth.

However, recursive CTE’s only allow one recursion level. We cannot nest the WITH clause.

To work around that, we can use PostgreSQL’s ability to run set-returning functions recursively. We will use the function-based recursion to iterate the parent-child axis, and the CTE-based recursion to iterate siblings axis.

We need to create a function that would take a node’s id on input and return a set of its children on output, with the function recursively applied to each of the children. To find a set of children, we will implement a recursive CTE that finds the first child in the anchor part and the next sibling in the recursive part.

Here’s the function:

CREATE OR REPLACE FUNCTION fn_get_children(id INT, level INT)
RETURNS SETOF INT[] AS
$$
        WITH    RECURSIVE q AS
                (
                SELECT  (hc).id, (hc).lft, (hc).rgt, prgt
                FROM    (
                        SELECT  (
                                SELECT  hc
                                FROM    t_hierarchy hc
                                WHERE   hc.lft > hp.lft
                                        AND hc.lft < hp.rgt
                                ORDER BY
                                        hc.lft
                                LIMIT 1
                                ) hc,
                                rgt AS prgt
                        FROM    t_hierarchy hp
                        WHERE   hp.id = $1
                        ) q2
                UNION ALL
                SELECT  (hc).id, (hc).lft, (hc).rgt, prgt
                FROM    (
                        SELECT  (
                                SELECT  hc
                                FROM    t_hierarchy hc
                                WHERE   hc.lft > q.rgt
                                        AND hc.lft < q.prgt
                                ORDER BY
                                        hc.lft
                                LIMIT 1
                                ) hc,
                                prgt
                        FROM    q
                        WHERE   q.lft IS NOT NULL
                        ) q2
                )
        SELECT  CASE which
                WHEN 1 THEN ARRAY[q.id, $2]
                ELSE fn_get_children(q.id, $2 - 1)
                END
        FROM    (
                VALUES (1), (2)
                ) vals(which)
        CROSS JOIN
                q
        WHERE   q.id IS NOT NULL
                AND $2 > 0
        ORDER BY
                id, which;
$$ LANGUAGE sql;

The function accepts a node’s id and a level on input, and returns a set of arrays, each corresponding to one of the node’s children and its level. The level returned by the function decreases and in fact represents not the level as such, but the number of levels left to reach the filter-set bottom. But since the initial level is user-set, it is easy to cast it to the actual level.

Let’s run the function:

SELECT  c[1], 3 - c[2]
FROM    fn_get_children(1, 2) c;
c ?column?
6 1
34 2
35 2
33 2
31 2
32 2
7 1
38 2
40 2
39 2
36 2
37 2
8 1
41 2
42 2
43 2
44 2
45 2
9 1
47 2
50 2
46 2
48 2
49 2
10 1
53 2
55 2
51 2
52 2
54 2
30 rows fetched in 0.0023s (0.0658s) 
Function Scan on fn_get_children c  (cost=0.00..262.50 rows=1000 width=32)

As we can see, the function returned all children and grandchildren of the node 1 along with their level, and did it in only 65 ms.

]]> http://explainextended.com/2010/03/02/postgresql-using-recursive-functions-in-nested-sets/feed/ 0 PostgreSQL: nested sets and R-Tree http://explainextended.com/2010/03/01/postgresql-nested-sets-and-r-tree/ http://explainextended.com/2010/03/01/postgresql-nested-sets-and-r-tree/#comments Mon, 01 Mar 2010 20:00:04 +0000 Quassnoi http://explainextended.com/?p=4506 A feedback on one of my previous articles comparing adjacency list and nested sets models for PostgreSQL.

Jay writes:

In your series on adjacency lists vs nested sets, you discuss geometric types and R-Tree indexes in MySQL, but you don’t discuss them when discussing the same subject with PostgreSQL, which also has geometric types and R-Tree indexing (mostly available through GiST indexes).

To make it simple I added the following line after the data insertion part of the script at Nested Sets – Postgresql:

ALTER TABLE t_hierarchy ADD COLUMN sets POLYGON;
UPDATE t_hierarchy SET sets = POLYGON(BOX(POINT(-1,lft), POINT(1, rgt)));

It needed to be a POLYGON instead of a BOX since there is a @>(POLYGON,POLYGON) function but no @>(BOX,BOX) function, and the polygon was cast from the box to create the rectangle shape required.

It outperforms the adjacency list on all descendants; outperforms it on all ancestors (not by much); performs reasonably well on all descendants up to a certain level on items with few descendants (e. g. 31415) and badly on items with many descendants (e. g. 42).

It still completes in less than 20 seconds though, which is an improvement over 1 minute.

PostgreSQL does support R-Tree indexes indeed (through GiST interface), and they can be used to improve the efficiency of the nested sets model.

Let’s create a sample table and try some of the queries that Jay proposed:

Table creation details

To make the management of the table easier, I didn’t create an additional column with the geometric representation of the nested sets, but instead just defined an index on a derived expression, so that updating lft and rgt columns would be enough to update the set.

Now, let’s see how these queries perform.

All descendants

SELECT  SUM(LENGTH(hc.stuffing)), COUNT(*)
FROM    t_hierarchy hp
JOIN    t_hierarchy hc
ON      POLYGON(BOX(POINT(-1, hc.lft), POINT(1, hc.rgt))) <@ POLYGON(BOX(POINT(-1, hp.lft), POINT(1, hp.rgt)))
WHERE   hp.id = 42

View query details

Quite fast, 213 ms.

All ancestors

SELECT  hc.id, hc.lft, hc.rgt, hc.parent
FROM    t_hierarchy hp
JOIN    t_hierarchy hc
ON      POLYGON(BOX(POINT(-1, hc.lft), POINT(1, hc.rgt))) @> POLYGON(BOX(POINT(-1, hp.lft), POINT(1, hp.rgt)))
WHERE   hp.id = 42

View query details

Extremely fast: only 10 ms.

All descendants up to a certain level

SELECT  hc.id, hc.lft, hc.rgt, hc.parent
FROM    t_hierarchy hp
JOIN    t_hierarchy hc
ON      POLYGON(BOX(POINT(-1, hc.lft), POINT(1, hc.rgt))) <@ POLYGON(BOX(POINT(-1, hp.lft), POINT(1, hp.rgt)))
WHERE   hp.id = 42
        AND
        (
        SELECT  COUNT(*)
        FROM    t_hierarchy hcp
        WHERE   POLYGON(BOX(POINT(-1, hc.lft), POINT(1, hc.rgt))) <@ POLYGON(BOX(POINT(-1, hcp.lft), POINT(1, hcp.rgt)))
        ) -
        (
        SELECT  COUNT(*)
        FROM    t_hierarchy hpp
        WHERE   POLYGON(BOX(POINT(-1, hp.lft), POINT(1, hp.rgt))) <@ POLYGON(BOX(POINT(-1, hpp.lft), POINT(1, hpp.rgt)))
        ) <= 2

View query details

This, exactly as was mentioned by Jay, is much faster than using a B-Tree index but still too slow: 20 seconds.

Analysis

The nested sets model, improved by using the R-Tree indexes, provides a way to tell if two records are in the same ancestry chain.

However, even with the R-Tree, the model provides no simple means to tell how deep is a record nested.

To check it, an R-Tree index scan should be made which would return all of the record’s ancestors, the the number of the ancestors is to be compared with that of the parent node.

For a record with lots of ancestors (which was the case for the record 42 we used in the test queries), this means that thousands of records should be checked in a nested loop, out of which only a dozen will be returned.

Ironically, for the real-world models, this type of query is most often used, and used against the records with lots of descendants it is.

Usually, when hierarchical data are stored in a database, they are presented to a user in the form of a tree view. When the user opens the catalog, the first-level entries are show; when the user clicks on expand button of an entry, all immediate children of the entry should be shown.

Since users usually start browsing from the beginning, clicking the expand buttons on the first-level or second-level entries is what happens most often. And, unfortunately, it takes the most time to execute these queries.

Adjacency list model provides a constant time solution to this problem, since fetching all immediate children requires a single index scan. This is extremely fast on showing the immediate children.

A user can also click on expand all which should just return all children of the given entry.

However, clicking on expand all on a high-level entry will return too many records, so a time to download them or represent them in the GUI will be much more than that required to fetch them out of the table. A properly written GUI usually limits the level of the records returned so that GUI remains responsive, which, it its turn, implies the same problem of filtering on level.

The low-level entries (for which it makes sense to implement expand all without any limitations) can be queried for their descendants with the R-Tree query in the nested sets model or with a recursive query in the adjacency list model almost equally fast, since low-level entries contain few records.

The same applies to selecting all ancestors. Despite the fact that the nested sets model outperforms slightly the adjacency list model on this type of query, the absolute numbers are very small and the times that both queries take are almost imperceptible to the bare eye. A hierarchy is seldom more than a dozen levels deep, and fetching each ancestor even with a recursive query requires but one unique index scan per ancestor.

However, one may still be forced to use the nested tree model. This may be the way an ORM stores its data in the database; a heavily used legacy schema too old and scary to touch; or just some obscure model which mostly requires fetching all descendants fast with an occasional need to filter on the level.

Here are some methods to deal with it.

Analytic functions

Though there is no efficient way to filter all descendants on the level, there still is a way to fetch all immediate children of a record.

If we select all records within lft and rgt of a given entry and order them by lft, the first record will be the first immediate child of the entry.

All descendants of the first child will be returned before the second child and have rgt less than that of the first child.

This means that if we record the MAX(rgt) fetched so far, it will be that of the last immediate child of the entry fetched so far:

id lft rgt MAX(rgt)
2 2 11 11
3 3 4 11
4 5 8 11
5 6 7 11
6 9 10 11
7 12 15 15
8 13 14 15

This means that each value of MAX(rgt) will correspond to exactly one immediate child; and the first entry in the recordset holding the value of MAX(rgt) will be that first child.

Several method exist to select records holding group-wise maximum in PostgreSQL. In this case, is will be best to use PostgreSQL’s DISTINCT ON.

Here’s the query:

SELECT  DISTINCT ON (MAX(hc.rgt) OVER (ORDER BY hc.lft)) hc.id, hc.lft, hc.rgt
FROM    t_hierarchy hp
JOIN    t_hierarchy hc
ON      hc.lft > hp.lft
        AND hc.lft < hp.rgt
WHERE   hp.id = 42
ORDER BY
        MAX(hc.rgt) OVER (ORDER BY hc.lft), hc.lft

View query details

, which is reasonably fast (only 160 ms).

Using PostgreSQL 8.4 recursive abilities, this approach can be extended to select the descendants up to any level (provided as a parameter to the query).

Here’s the query to select all children and grandchildren:

WITH    RECURSIVE
        q AS
        (
        SELECT  id, lft, rgt, 1 AS lvl
        FROM    t_hierarchy
        WHERE   id = 42
        UNION ALL
        SELECT  *
        FROM    (
                SELECT  DISTINCT ON (MAX(hc.rgt) OVER (PARTITION BY q.id ORDER BY hc.lft)) hc.id, hc.lft, hc.rgt, lvl + 1
                FROM    q
                JOIN    t_hierarchy hc
                ON      hc.lft > q.lft
                        AND hc.lft < q.rgt
                WHERE   lvl <= 2
                ORDER BY
                        MAX(hc.rgt) OVER (PARTITION BY q.id ORDER BY hc.lft), hc.lft
                ) q2
        )
SELECT  *
FROM    q

View query details

This is also reasonably fast, only 432 ms. It is slower than the same adjacency list query (which completes in several milliseconds), but still is much faster than R-Tree and of course the least efficient B-Tree solutions involving COUNT(*) and can ease your life if you have to deal with a nested sets model.

To be continued.

]]> http://explainextended.com/2010/03/01/postgresql-nested-sets-and-r-tree/feed/ 0 Six degrees of separation http://explainextended.com/2010/02/27/six-degrees-of-separation/ http://explainextended.com/2010/02/27/six-degrees-of-separation/#comments Sat, 27 Feb 2010 20:00:45 +0000 Quassnoi http://explainextended.com/?p=4477 Answering questions asked on the site.

Kathy asks:

I am developing a social network site in PostgreSQL and want to find out if two people are no more than 6 friends apart.

If your site grows popular, most probably, they are not. But we better check.

On most social networks, friendship is a symmetric relationship (however, LiveJournal is a notable exception). This means that if Alice is a friend to Bob, then Bob is a friend to Alice as well.

The friendship relationship is best stored in a many-to-many link table with a PRIMARY KEY on both link fields and an additional check condition: the friend with the least id should be stored in the first column. This is to avoid storing a relationship twice: a PRIMARY KEY won’t be violated if the same record with the columns swapped will be inserted, but the check constraint will. The check constraint will also forbid storing a friend relationship to itself.

Let’s create a sample table:

Table creation details

This table stores records for 1,000,000 people having 20 friends each in average. The first column is named orestes and the second one pylades.

With new PostgreSQL 8.4 it is easy to write a recursive query that would traverse the relationship graph up to the given level and stop on the first match.

However, each recursion step requires a join, and as the number of records in the input recordset for the recusion grows with level, the joins become less and less efficient. The number of records grows exponentially, and on level 6 there will be about 20 ^ 6 = 64,000,000 records on input. This is just too much for a join with a 20,000,000 records table.

As the chain length increases, the tree diverges, the number of the records grows and it becomes more and more costly to join them with the table.

To work around this, we should use Bogdan the tunnel builder’s algorithm.

British and French governments submit a tender to build a tunnel under the Channel. Many companies apply, all demanding years of time and billions pounds of money, so their offers are refused.

One day, Bogdan drops in and offers his services.

How much money would you demand for your work?, the official asks. Me and my brother Roman are good eaters, so we will need to buy a decent meal every day. 50 pounds a day will be OK.

That’s pretty cheap; and how much time will you need? Me and my brother Roman are fast diggers; one mile a day I think we will dig.

Oh, that’s pretty fast! But how will you be able to work on such a low budget in such a short time?, the official asks out of curiosity.

That’s simple, Bogdan answers, I start digging from the British coast, my brother Roman starts digging from the French coast; the moment we meet, the work is over.

OK, says the official, but what if you don’t meet?.

No worries then: you get two tunnels for the price of one

We should do a similar thing here. Instead of traversing the 6 levels from the beginning, we will traverse just 3 levels from each side, then join the resulting recordsets and hope some matching records will be found.

Traversing only 3 levels will be quite fast; and the resulting recordsets will be of moderate size so joining them will be easy.

As an extra, we will return the shortest path from one person to the other. To do this, we will need to record the friendship chain in an array. PostgreSQL does not offer an easy way to reverse an array, so in the first recordset, we will append the friends to the array, while in the second one we will prepend them. This way, we should just concatenate the resulting tunnelsarrays.

Here’s the query:

WITH    RECURSIVE
        q1 (person, chain, lvl) AS
        (
        SELECT  123456, ARRAY[123456], 1
        UNION ALL
        SELECT  friend, chain || friend, lvl + 1
        FROM    (
                SELECT  q1.*,
                        friend
                FROM    q1
                JOIN    (
                        SELECT  orestes AS me, pylades AS friend
                        FROM    friends
                        UNION ALL
                        SELECT  pylades AS me, orestes AS friend
                        FROM    friends
                        ) f
                ON      person = me
                WHERE   lvl <= 3
                ) qo
        ),
        q2 (person, chain, lvl) AS
        (
        SELECT  654321, ARRAY[654321], 1
        UNION ALL
        SELECT  friend, friend || chain, lvl + 1
        FROM    (
                SELECT  q2.*,
                        friend
                FROM    q2
                JOIN    (
                        SELECT  orestes AS me, pylades AS friend
                        FROM    friends
                        UNION ALL
                        SELECT  pylades AS me, orestes AS friend
                        FROM    friends
                        ) f
                ON      person = me
                WHERE   lvl <= 3
                ) qo
        )
SELECT  (q1.chain || q2.chain[2:q2.lvl])::TEXT AS chain
FROM    q1
JOIN    q2
ON      q2.person = q1.person
ORDER BY
        q1.lvl + q2.lvl
LIMIT 1

View query details

Note that the anchor part can not be used more than once in a recursive expression. To work around that, we had to join it to a derived table (a UNION ALL of two copies of the table with the columns swapped). However, PostgreSQL’s optimizer was smart enough to push the join predicate into the derived table and distribute the queries so that each part uses a corresponding index efficiently. This helps to traverse the tree and build the recordsets from both ends.

Each of the recordsets has only about 8,000 records, so scanning and joining them is very fast.

The whole query takes just a little longer than 0.5 seconds.

Hope that helps.

I’m always glad to answer the questions regarding database queries.

Ask me a question

]]> http://explainextended.com/2010/02/27/six-degrees-of-separation/feed/ 0 Efficient circle distance testing http://explainextended.com/2010/02/26/efficient-circle-distance-testing/ http://explainextended.com/2010/02/26/efficient-circle-distance-testing/#comments Fri, 26 Feb 2010 20:00:41 +0000 Quassnoi http://explainextended.com/?p=4449 Answering questions asked on the site.

eptil asks:

I am using SQL Server 2008, but not the spatial features.

I have a table with few entries, only 40,000. There is an id INT PRIMARY KEY column and two columns storing a 2d coordinate, both decimals.

I would like to find all the records that do not have other records within a given radius. The query I am using at the moment is:

SELECT  id, x, y
FROM    mytable t1
WHERE   (
        SELECT  COUNT(*)
        FROM    mytable t2
        WHERE   ABS(t1.x - t2.x) < 25
                AND ABS(t1.y - t2.y) < 25
        ) = 1


This is taking 15 minutes to run at times.

Is there a better way?

Of course using spatial abilities would be a better way, but it is possible to make do with plain SQL. This will also work in SQL Server 2005.

In most database engines, the spatial indexes are implemented as the R-Tree structures. SQL Server, however, uses another approach: surface tesselation.

Basically, it divides the surface into a finite number of tiles, each assigned with a unique number. The identifiers of tiles covered by the object are stored as keys of a plain B-Tree index.

When SQL Server’s optimizer sees a geometrical predicate against an indexed column, it calculates the numbers of tiles that possibly can satisfy this predicate. Say, if the tiles are defined as squares with side 1, the predicate column.STDistance(@mypoint) < 2 can only be satisfied by the objects within 2 tiles away from @mypoint’s tile. This gives a square of 25 tiles with @mypoint’s tile in the center. The tile numbers can be found and searched for using the index. Exact filtering condition is then applied to each candidate value returned by the index.

Same solution can be used in our case even without the spatial functions. Comparing tile numbers is an equijoin and hash join method is eligible for this operation. We can even choose the tiling algorithm individually for each query, since we don’t have to store the tile identifiers in the table, and the hash table will be built dynamically anyway.

Let’s create a sample table and see how it works:

CREATE SCHEMA [20100226_circle]
CREATE TABLE t_circle (
        id INT NOT NULL PRIMARY KEY,
        x DECIMAL (15, 3) NOT NULL,
        y DECIMAL (15, 3) NOT NULL,
        )
GO
BEGIN TRANSACTION
SELECT  RAND(20100226)
DECLARE @cnt INT
SET @cnt = 1
WHILE @cnt <= 50000
BEGIN
        INSERT
        INTO    [20100226_circle].t_circle (id, x, y)
        VALUES  (
                @cnt,
                RAND() * 3200,
                RAND() * 3200
                )
        SET @cnt = @cnt + 1
END
COMMIT
GO

The table contains 50,000 points on random places within a square of 3,200 × 3,200 units.

We can optimize the original query a little by using NOT EXISTS instead of COUNT(*):

SELECT  *
FROM    [20100226_circle].t_circle co
WHERE   NOT EXISTS
        (
        SELECT  NULL
        FROM    [20100226_circle].t_circle ci
        WHERE   SQRT(POWER(co.x - ci.x, 2) + POWER(co.y - ci.y, 2)) < 25
                AND co.id <> ci.id
        )
ORDER BY
        id

View query results

, but this would still be quite slow.

The problem is that the nested loops go nowhere: they are still inside the plan, but return earlier. As a result, the query takes 5 minutes instead of 15, which is still too much.

To improve the query we need to make an efficient anti-join method to work, and the tesselation strategy is a way to go. Here’s what we need to do to implement this strategy:

  • Tesselate the surface and assign a unique number to each tile. Since we need to search for the records within 25 units, it will be a reasonable idea to divide the surface into a number of squares 25 × 25 units in size, numbered column-wise. To find out the number of rows and columns, we should just find the MIN and MAX x and y.

    This is the sample tesselation, assuming that MIN(x) and MIN(y) are 25 × 100 = 2500 units apart (and same with y).

  • Find the tile each point belongs to.

  • Build a recordset that would correspond each point to each of the tiles the neighbors can theoretically reside in. Since a circle with radius of 25 units theoretically may cover up to 9 adjacent tiles, each tile should be corresponded to each of the 9 tiles forming a 3 × 3 square with a unit’s tile in the center.

  • For each point, make sure that no other points exists within the neighboring tiles, additionally applying a fine-filtering condition.

    This may sound redundant, since the point-neighbor combination is unique, as well as point-tile, so only one of the 9 candidates will satisfy the join, even if the points reside in the adjacent tiles. But finding the exact distance requires a complex expression while comparing the tile numbers is an equality correlation and as such is eligible by an efficient anti-join method like HASH ANTI JOIN. The coarse filtering on tiles will sieve out most of the far neighbors so that the only adjacent neighbors will require special attention.

And here’s the query:

WITH    extremes AS
        (
        SELECT  *,
                maxx - minx AS width,
                maxy - miny AS height
        FROM    (
                SELECT  FLOOR(MIN(x) / 25) AS minx,
                        CEILING(MAX(x) / 25) AS maxx,
                        FLOOR(MIN(y) / 25) AS miny,
                        CEILING(MAX(y) / 25) AS maxy
                FROM    [20100226_circle].t_circle
                ) q
        ),
        tileset (dim) AS
        (
        SELECT  -1
        UNION ALL
        SELECT  0
        UNION ALL
        SELECT  1
        ),
        tiles AS
        (
        SELECT  id, x, y, minx, miny, width,
                (FLOOR(x / 25) - minx) * width + FLOOR(y / 25) - miny AS tile
        FROM    extremes
        CROSS JOIN
                [20100226_circle].t_circle
        ),
        neighbors AS
        (
        SELECT  ti.*,
                (FLOOR(ti.x / 25) - ti.minx + nx.dim) * ti.width +
                FLOOR(ti.y / 25) - ti.miny + ny.dim AS mtile
        FROM    tiles ti
        CROSS JOIN
                tileset nx
        CROSS JOIN
                tileset ny
        )
SELECT  *
FROM    tiles tn
WHERE   NOT EXISTS
        (
        SELECT  NULL
        FROM    neighbors n
        WHERE   n.mtile = tn.tile
                AND n.id <> tn.id
                AND SQRT(SQUARE(n.x - tn.x) + SQUARE(n.y - tn.y)) < 25
        )
ORDER BY
        id
id x y minx miny width tile
205 2247.896 3198.399 0 0 128 11519
2867 2159.626 1120.590 0 0 128 11052
13644 4.951 3165.734 0 0 128 126
15917 2747.826 3041.280 0 0 128 14073
25183 1858.866 326.416 0 0 128 9485
43211 1176.369 98.079 0 0 128 6019
6 rows fetched in 0.0017s (3.0781s) 
Table 't_circle'. Scan count 11, logical reads 3087, physical reads 0, read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.
Table 'Worktable'. Scan count 2, logical reads 207406, physical reads 0, read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.
Table 'Worktable'. Scan count 0, logical reads 0, physical reads 0, read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0. 

SQL Server Execution Times:
   CPU time = 4235 ms,  elapsed time = 3068 ms.

  |--Parallelism(Gather Streams, ORDER BY:([test].[20100226_circle].[t_circle].[id] ASC))
       |--Sort(ORDER BY:([test].[20100226_circle].[t_circle].[id] ASC))
            |--Compute Scalar(DEFINE:([Expr1007]=floor([Expr1003]/(25.)), [Expr1009]=floor([Expr1005]/(25.)), [Expr1011]=ceiling([Expr1004]/(25.))-floor([Expr1003]/(25.)), [Expr1016]=(([Expr1044]-floor([Expr1003]/(25.)))*(ceiling([Expr1004]/(25.))-floor([Expr1003]/(25.)))+[Expr1045])-floor([Expr1005]/(25.))))
                 |--Hash Match(Left Anti Semi Join, HASH:([Expr1055])=([Expr1054]), RESIDUAL:([Expr1054]=[Expr1055] AND [test].[20100226_circle].[t_circle].[id]<>[test].[20100226_circle].[t_circle].[id] AND sqrt(square(CONVERT_IMPLICIT(float(53),[test].[20100226_circle].[t_circle].[x]-[test].[20100226_circle].[t_circle].[x],0))+square(CONVERT_IMPLICIT(float(53),[test].[20100226_circle].[t_circle].[y]-[test].[20100226_circle].[t_circle].[y],0)))<(2.500000000000000e+001)))
                      |--Bitmap(HASH:([Expr1055]), DEFINE:([Bitmap1056]))
                      |    |--Parallelism(Repartition Streams, Hash Partitioning, PARTITION COLUMNS:([Expr1055]))
                      |         |--Compute Scalar(DEFINE:([Expr1055]=(([Expr1044]-floor([Expr1003]/(25.)))*(ceiling([Expr1004]/(25.))-floor([Expr1003]/(25.)))+[Expr1045])-floor([Expr1005]/(25.))))
                      |              |--Nested Loops(Inner Join)
                      |                   |--Parallelism(Distribute Streams, Broadcast Partitioning)
                      |                   |    |--Stream Aggregate(DEFINE:([Expr1003]=MIN([partialagg1048]), [Expr1004]=MAX([partialagg1049]), [Expr1005]=MIN([partialagg1050])))
                      |                   |         |--Parallelism(Gather Streams)
                      |                   |              |--Stream Aggregate(DEFINE:([partialagg1048]=MIN([test].[20100226_circle].[t_circle].[x]), [partialagg1049]=MAX([test].[20100226_circle].[t_circle].[x]), [partialagg1050]=MIN([test].[20100226_circle].[t_circle].[y])))
                      |                   |                   |--Clustered Index Scan(OBJECT:([test].[20100226_circle].[t_circle].[PK__t_circle__62065FF3]))
                      |                   |--Compute Scalar(DEFINE:([Expr1044]=floor([test].[20100226_circle].[t_circle].[x]/(25.)), [Expr1045]=floor([test].[20100226_circle].[t_circle].[y]/(25.))))
                      |                        |--Clustered Index Scan(OBJECT:([test].[20100226_circle].[t_circle].[PK__t_circle__62065FF3]))
                      |--Parallelism(Repartition Streams, Hash Partitioning, PARTITION COLUMNS:([Expr1054]), WHERE:(PROBE([Bitmap1056])=TRUE))
                           |--Compute Scalar(DEFINE:([Expr1054]=(((([Expr1046]-floor([Expr1020]/(25.)))+CONVERT_IMPLICIT(decimal(10,0),[Union1037],0))*(ceiling([Expr1021]/(25.))-floor([Expr1020]/(25.)))+[Expr1047])-floor([Expr1022]/(25.)))+CONVERT_IMPLICIT(decimal(10,0),[Union1041],0)))
                                |--Nested Loops(Inner Join)
                                     |--Nested Loops(Inner Join)
                                     |    |--Parallelism(Distribute Streams, RoundRobin Partitioning)
                                     |    |    |--Nested Loops(Inner Join)
                                     |    |         |--Stream Aggregate(DEFINE:([Expr1020]=MIN([partialagg1051]), [Expr1021]=MAX([partialagg1052]), [Expr1022]=MIN([partialagg1053])))
                                     |    |         |    |--Parallelism(Gather Streams)
                                     |    |         |         |--Stream Aggregate(DEFINE:([partialagg1051]=MIN([test].[20100226_circle].[t_circle].[x]), [partialagg1052]=MAX([test].[20100226_circle].[t_circle].[x]), [partialagg1053]=MIN([test].[20100226_circle].[t_circle].[y])))
                                     |    |         |              |--Clustered Index Scan(OBJECT:([test].[20100226_circle].[t_circle].[PK__t_circle__62065FF3]))
                                     |    |         |--Constant Scan(VALUES:(((-1)),((0)),((1))))
                                     |    |--Constant Scan(VALUES:(((-1)),((0)),((1))))
                                     |--Table Spool
                                          |--Compute Scalar(DEFINE:([Expr1046]=floor([test].[20100226_circle].[t_circle].[x]/(25.)), [Expr1047]=floor([test].[20100226_circle].[t_circle].[y]/(25.))))
                                               |--Clustered Index Scan(OBJECT:([test].[20100226_circle].[t_circle].[PK__t_circle__62065FF3]))

This query only takes 3 seconds.

Hope that helps.

I’m always glad to answer the questions regarding database queries.

Ask me a question

]]> http://explainextended.com/2010/02/26/efficient-circle-distance-testing/feed/ 0 Matching sets: aggregates vs. first miss http://explainextended.com/2010/02/25/matching-sets-aggregates-vs-first-miss/ http://explainextended.com/2010/02/25/matching-sets-aggregates-vs-first-miss/#comments Thu, 25 Feb 2010 20:00:47 +0000 Quassnoi http://explainextended.com/?p=4436 From Stack Overflow:

Here is my schema:

Suppliers
sid sname address
Parts
pid pname color
Catalog
sid pid cost

I need to find the sids of suppliers who supply every red part or every green part.

This task requires matching the sets.

We need to compare two sets here: the first one is the set of the parts of given color; the second one is the set of parts provided by a given supplier. The former should be the subset of the latter.

Unlike other engines, MySQL does not provide the set operators like EXCEPT or MINUS that allow to check the subset / superset relationship very easily. We have to use the record-based solutions.

There are two ways to check that:

  • First miss technique: test each record from the subset candidate against the superset candidate, returning FALSE if there is no match.
  • Aggregate technique: compare the number of records in the subset candidate to the number of records in their intersection. If the numbers are equal, the sets match

Let’s test which way is faster in which cases. To do this, we will need some sample tables:

Table creation details

There are 10,000 suppliers and 2,000 parts.

The parts can be red, green or blue. As for red and green, there are 10 parts of each color, and they are distributed evenly across the suppliers. With blue, the situation is different: there are 1980 blue parts and only 200 first suppliers provide it. However, for each of the blue part suppliers, the probability of each blue part to be available is very high.

First miss

The first miss is a combination of NOT IN / NOT EXISTS clauses that immediately return FALSE whenever a single non-matching record is found. Since MySQL can only do nested loops, the performance of these queries is heavily dependent on proper indexing.

Let’s run this query to search for red or green parts:

SELECT  *
FROM    suppliers s
WHERE   EXISTS
        (
        SELECT  NULL
        FROM    (
                SELECT  'red' AS color
                UNION ALL
                SELECT  'green' AS color
                ) ci
        WHERE   color NOT IN
                (
                SELECT  color
                FROM    parts p
                WHERE   p.pid NOT IN
                        (
                        SELECT  pid
                        FROM    catalog c
                        WHERE   c.sid = s.sid
                        )
                )
        )

View query details

This query runs for 2.17 s.

The same query for the blue parts:

SELECT  *
FROM    suppliers s
WHERE   EXISTS
        (
        SELECT  NULL
        FROM    (
                SELECT  'blue' AS color
                ) ci
        WHERE   color NOT IN
                (
                SELECT  color
                FROM    parts p
                WHERE   p.pid NOT IN
                        (
                        SELECT  pid
                        FROM    catalog c
                        WHERE   c.sid = s.sid
                        )
                )
        )

View query details

This time the query is a little bit slower: 2.43 s.

Aggregate

In the aggregate solution, we first calculate the total number of parts for each color and the count the parts of this color supplied by each supplier. If the counts match, the sets match too.

Let’s try it on red and green parts first:

SELECT  c.sid, p.color
FROM    (
        SELECT  color, COUNT(*) AS total
        FROM    parts
        WHERE   color IN ('red', 'green')
        GROUP BY
                color
        ) t
JOIN    parts p
ON      p.color = t.color
JOIN    catalog c
ON      c.pid = p.pid
GROUP BY
        sid, color, total
HAVING  COUNT(*) = total

View query details

And the same query on the blue parts:

SELECT  c.sid, p.color
FROM    (
        SELECT  color, COUNT(*) AS total
        FROM    parts
        WHERE   color IN ('blue')
        GROUP BY
                color
        ) t
JOIN    parts p
ON      p.color = t.color
JOIN    catalog c
ON      c.pid = p.pid
GROUP BY
        sid, color, total
HAVING  COUNT(*) = total

View query details

We see that the queries complete in 1.14 seconds and 3.09 seconds, accordingly.

The aggregate method is more efficient for the red and green parts, while the first miss is more efficient for the blue parts.

Analysis

What is the reason of such a difference in performance?

The first miss method generally needs to parse less records (only those before the first miss), but each record is searched for in a nested loop, starting from the index root. The aggregate method needs to parse all records to calculate the COUNT(*), but these records are fetched in a sequential index access.

The red and green parts have the large number of small sets, with at most 10 records in each. The probability of the miss is relatively small: a large number of records should be parsed before any record is missed in a set. The aggregates, on the other hand, can be calculated very fast, since there are relatively few records to aggregate.

With the blue parts, the situation is different. There are few large sets, and calculating the aggregates requires fetching, sorting and grouping of too many records. First misses, on the other hand, occurs almost instantly: the vast majority of the suppliers do not offer any blue parts at all.

Summary

As it often happens, the performance of the two methods to compare the sets depends on the data distribution.

The sets with less records and lower probability of the record miss will benefit from the aggregate method, since the performance increase caused by the sequential access to the records overweights the need to parse a larger number of the records.

The sets with more records and higher probability of a miss will return the misses very soon, so the first miss method is more beneficial for them.

]]> http://explainextended.com/2010/02/25/matching-sets-aggregates-vs-first-miss/feed/ 0 Sargability of monotonic functions: example http://explainextended.com/2010/02/23/sargability-of-monotonic-functions-example/ http://explainextended.com/2010/02/23/sargability-of-monotonic-functions-example/#comments Tue, 23 Feb 2010 20:00:25 +0000 Quassnoi http://explainextended.com/?p=4407 In my previous article I presented a proposal to add sargability of monotonic functions into the SQL engines.

In a nutshell: a monotonic function is a function that preserves the order of the argument so that it gives the larger results for the larger values of the argument. It is easy to prove that a B-tree with each key replaced by the result of the function will remain the valid B-Tree and hence can be used to search for ranges of function results just like it is used to search for ranges of values.

With a little effort, a B-Tree can also be used to search for the ranges of piecewise monotonic functions: those whose domain can be split into a number of continuous pieces with the function being monotonic within each piece (but it may be not monotonic and even not continuous across the pieces).

In this article, I’ll demonstrate the algorithm to do that (implemented in pure SQL on PostgreSQL).

I will show how the performance of simple query 

SELECT  *
FROM    t_sine
WHERE   SIN(value) BETWEEN 0.1234 AND 0.1235

could be improved if the sargability of monotonic functions had been implemented in the optimizer.

To do this, I will create a sample table:

Table creation details

This table contains 1,000,000 records with value randomly distributed from 0 to 101.

To select the records we need, we can use a very simple and straightforward query:

SELECT  *
FROM    t_sine
WHERE   SIN(value) BETWEEN 0.4452 AND 0.4453

View query details

which returns 31 records in 3.25 seconds.

The query uses a full table scan with the filter applied to each record.

Let’s try to improve it.

Function description

According to the notation I proposed in the previous article, the monotony of the function SIN() should be described as this:

SIN(arg FLOAT) MONOTONIC PIECEWISE
DEFINED BY FLOOR(arg / PI() + 0.5)
CASE PIECE % 2
WHEN 0 THEN DECREASING INVERSE PIECE * PI() + ASIN(RESULT)
ELSE INCREASING INVERSE PIECE * PI() - ASIN(RESULT)
END

This means that:

  1. The function is piecewise monotonic,

  2. The pieces are defined by the function FLOOR(arg / PI() + 0.5) (which essentially returns the number of the half-wave the argument belongs too),

  3. The function monotony varies depending on the piece,

  4. On odd pieces, the function increases,

  5. On even pieces, the function decreases.

  6. A single inverse expression is provided for each monotony

Note that mathematically the function is strictly monotonic on each of its pieces. However, due to the rounding errors, different arguments can yield same function results, so the function value may map back to a range of the arguments rather than a single value.

In theory, it is possible to write a single expression which would map the function’s result to the pair of values defining the beginning and the end of such a range. However, the expression would be quite complex. So for illustration purposes I’ll make do with a single inverse function that yields an approximation of the back mapping. To find the exact range, some extra effort will be required.

Building the pieces

The function is piecewise monotonic and the pieces are defined by a function. For the pieces to be continuous, the function that defines them should be itself monotonic over all its domain.

The function that defines the pieces is FLOOR(arg / PI() + 0.5).

It is a superposition of the three functions:

  • OPERATOR_DIVISION(arg1 FLOAT, arg2 FLOAT)
    MONOTONIC OVER (arg1)
    CASE WHEN arg2 > 0 THEN INCREASING INVERSE RESULT * arg2
    WHEN arg2 = 0 THEN UNDEFINED
    WHEN arg2 < 0 THEN DECREASING INVERSE RESULT * arg2
    END

  • OPERATOR_PLUS(arg1 FLOAT, arg2 FLOAT) MONOTONIC
    OVER (arg1) STRICTLY INCREASING INVERSE RESULT - arg2,
    OVER (arg2) STRICTLY INCREASING INVERSE RESULT - arg1

  • FLOOR(arg FLOAT) MONOTONIC INCREASING
    INVERSE
    FROM RESULT EXACT
    TO RESULT + 1 EXACT EXCLUDE

which are, given the values of the constants provided in the secondary arguments, are increasing over the argument. As we know from math, a superposition of monotonic functions is also monotonic.

Each function is defined with a single inverse condition which maps the result of the function back to a value near the range of the arguments yielding the result. The exact range has to be sought for using index seek over (hopefully) not too many records.

Sequentially applying the inverse expressions of each of the constituent functions to the result of the piece defining function, we get the following inverse expression for the latter:

  1. FLOOR(OPERATOR_PLUS(OPERATOR_DIVISION(arg, PI()), 0.5)) = PIECE
  2. OPERATOR_PLUS(OPERATOR_DIVISION(arg, PI()), 0.5) ∈ [ PIECE, PIECE + 1 )
  3. OPERATOR_DIVISION(arg, PI()) ∈ [ ≈(PIECE - 0.5), ≈((PIECE + 1) - 0.5) ]
  4. arg ∈ [ ≈((PIECE - 0.5) * PI()), ≈(((PIECE + 1) - 0.5) * PI()) ]

For each piece, we how have a pair of values approximately defining the range of values belonging to the piece.

To find out the exact bounds, we need to do the following:

  1. Calculate the piece for the minimal value

  2. Find the approximate upper bound for the piece.

  3. Scanning the keys to the left, find the rightmost key to the left of the upper bound that belongs to the current (or previous) piece.

  4. Scanning the keys to the right, find the first key of the next piece.

  5. Scanning a single key to the left, find the last key of the current piece.

  6. Recursively repeat steps 1 to 5, taking the first value the next piece calculated on step 4 as a seed for the step 1, until step 4 fails (which means that the pieces are over).

This procedure guarantees that we always get the correct bounds even with the inexact inverse value, since it correctly handles both overflow and underflow of the inverse value, as show on the pictures below:

Overflow

Underflow

Here's a query that selects the first and the last key of each piece:

View the query

Despite being huge in size, the query is very efficient and completes in only 48 ms.

Locating values within the pieces

Now, when we have the exact bounds of each piece, we need to locate the records within each piece.

Since we don't have exact inverse function here, the basic idea is the same as above: given the approximate inverse, locate the exact bound using the iterative approach:

  1. Locate the first key to the left of the inverse which yields the function result less than the one sought for, up to the first key of the piece. Should this search fail, the first key of the piece is the lower bound.
  2. Locate the first key to the right of that found on the previous step that yields the function value equal to or greater than the one sought for, up to the last key of the piece. Return NULL should it fail

Since we have an inclusive range here, we don't need the third step (final scan to the left to find the rightmost least value) that we used when searching for the pieces.

This algorithm searches for the lower bound; to search for the upper bound, we just need to inverse both directions and tests (left becomes right, less becomes greater etc).

Since the monotony of the function varies from piece to piece, we should take this into account. For the pieces where the function's monotony is DECREASING we should swap the order of the bounds: the upper bound of the expression becomes the lower bound or the range of values and vice versa. This can be handled merely by substituting the conditions into the very same CASE expression that defines the monotony.

When we locate the upper and the lower bounds for each piece, we should just join t_sine on the following condition:

ON value BETWEEN llimit and ulimit

It can happen so that the lower bound found by the algorithm exceeds the upper bound. This is a perfectly normal situation meaning that no keys match the condition and the range diverged. BETWEEN predicate will handle this.

It also can happen that one of the bounds is a NULL. This is also a valid situation, meaning that no value within the piece exceeds the lower bound (or falls short of the upper one). BETWEEN will also take care of it.

Final query

And here's the final query. Be ready, you'll have to spin your mouse wheel a lot:

WITH    RECURSIVE
        d AS (
        SELECT  piece,
                minv,
                COALESCE(
                (
                SELECT  value
                FROM    t_sine
                WHERE   value < nv[1]
                ORDER BY
                        value DESC
                LIMIT 1
                ),
                (
                SELECT  MAX(value)
                FROM    t_sine
                )
                ) AS maxv,
                nv[1] AS nextv,
                nv[2] AS nextpiece
        FROM    (
                SELECT  minv, piece,
                        (
                        SELECT  ARRAY[value, FLOOR(value / PI() + 0.5)]
                        FROM    t_sine
                        WHERE   value >
                                (
                                SELECT  value
                                FROM    t_sine
                                WHERE   value <= ((piece + 1) - 0.5) * PI()
                                        AND FLOOR(value / PI() + 0.5) <= piece
                                ORDER BY
                                        value DESC
                                LIMIT 1
                                )
                        ORDER BY
                                value
                        LIMIT 1
                        ) nv
                FROM    (
                        SELECT  minv, FLOOR(minv / PI() + 0.5) AS piece
                        FROM    (
                                SELECT  MIN(value) AS minv
                                FROM    t_sine
                                ) q
                        ) q2
                ) q3
        UNION ALL
        SELECT  piece,
                minv,
                COALESCE(
                (
                SELECT  value
                FROM    t_sine
                WHERE   value < nv[1]
                ORDER BY
                        value DESC
                LIMIT 1
                ),
                (
                SELECT  MAX(value)
                FROM    t_sine
                )
                ) AS maxv,
                nv[1] AS nextv,
                nv[2] AS nextpiece
        FROM    (
                SELECT  minv, piece,
                        (
                        SELECT  ARRAY[value, FLOOR(value / PI() + 0.5)]
                        FROM    t_sine
                        WHERE   value >
                                (
                                SELECT  value
                                FROM    t_sine
                                WHERE   value <= ((piece + 1) - 0.5) * PI()
                                        AND FLOOR(value / PI() + 0.5) <= piece
                                ORDER BY
                                        value DESC
                                LIMIT 1
                                )
                        ORDER BY
                                value
                        LIMIT 1
                        ) nv
                FROM    (
                        SELECT  nextv AS minv, nextpiece AS piece
                        FROM    d
                        WHERE   nextpiece IS NOT NULL
                        ) q2
                ) q3
        )
SELECT  l.*, s.*, SIN(value)
FROM    (
        SELECT  minv, maxv,
                CASE piece::INTEGER % 2
                WHEN 0 THEN
                        (
                        SELECT  value
                        FROM    t_sine
                        WHERE   value >=
                                COALESCE(
                                (
                                SELECT  value
                                FROM    t_sine
                                WHERE   value <= LEAST(piece * PI() + ASIN(0.4452), maxv)
                                        AND value >= minv
                                        AND SIN(value) < 0.4452
                                ORDER BY
                                        value DESC
                                LIMIT 1
                                ),
                                minv
                                )
                                AND value <= maxv
                                AND SIN(value) >= 0.4452
                        ORDER BY
                                value
                        LIMIT 1
                        )
                ELSE
                        (
                        SELECT  value
                        FROM    t_sine
                        WHERE   value >=
                                COALESCE(
                                (
                                SELECT  value
                                FROM    t_sine
                                WHERE   value <= LEAST(piece * PI() - ASIN(0.4453), maxv)
                                        AND value >= minv
                                        AND SIN(value) > 0.4453
                                ORDER BY
                                        value DESC
                                LIMIT 1
                                ),
                                minv
                                )
                                AND value <= maxv
                                AND SIN(value) <= 0.4453
                        ORDER BY
                                value
                        LIMIT 1
                        )
                END AS llimit,
                CASE piece::INTEGER % 2
                WHEN 0 THEN
                        (
                        SELECT  value
                        FROM    t_sine
                        WHERE   value <=
                                COALESCE(
                                (
                                SELECT  value
                                FROM    t_sine
                                WHERE   value >= GREATEST(piece * PI() + ASIN(0.4453), minv)
                                        AND value <= maxv
                                        AND SIN(value) > 0.4453
                                ORDER BY
                                        value
                                LIMIT 1
                                ),
                                maxv
                                )
                                AND value >= minv
                                AND SIN(value) <= 0.4453
                        ORDER BY
                                value DESC
                        LIMIT 1
                        )
                ELSE
                        (
                        SELECT  value
                        FROM    t_sine
                        WHERE   value <=
                                COALESCE(
                                (
                                SELECT  value
                                FROM    t_sine
                                WHERE   value >= GREATEST(piece * PI() - ASIN(0.4452), minv)
                                        AND value <= maxv
                                        AND SIN(value) < 0.4452
                                ORDER BY
                                        value
                                LIMIT 1
                                ),
                                maxv
                                )
                                AND value >= minv
                                AND SIN(value) >= 0.4452
                        ORDER BY
                                value DESC
                        LIMIT 1
                        )
                END AS ulimit
        FROM    d
        ) l
JOIN    t_sine s
ON      value BETWEEN llimit AND ulimit

View query details

This 200-line monster completes in only 110 ms, or 30 times as fast as the original 3-liner:

SELECT  *
FROM    t_sine
WHERE   SIN(value) BETWEEN 0.1234 AND 0.1235

, yielding the same results.

Summary

This example was to demonstrate feasibility of the B-Tree indexes to be used in a search for the predicates involving monotonic functions and the performance gain achieved.

The performance gain is over 30 times for a table that fits completely into the cache, and will increase with the number of the cache misses increases.

The SQL implementation of the algorithm is in fact not optimal, since iterative searches for the value boundaries are implemented as the subqueries. Each subquery requires reentering the B-Tree and traversing it starting from the root. The native algorithm working within the optimizer could avoid this by caching the key position in the index and issuing next_key / prev_key commands, which would improve the algorithm yet more.

Sargability of the monotonic functions, as shown above, can help to make the queries like the one described in this article much more legible, maintainable and efficient.

]]> http://explainextended.com/2010/02/23/sargability-of-monotonic-functions-example/feed/ 0 Things SQL needs: sargability of monotonic functions http://explainextended.com/2010/02/19/things-sql-needs-sargability-of-monotonic-functions/ http://explainextended.com/2010/02/19/things-sql-needs-sargability-of-monotonic-functions/#comments Fri, 19 Feb 2010 20:00:28 +0000 Quassnoi http://explainextended.com/?p=4149

Image by ndevil

I’m going to write a series of articles about the things SQL needs to work faster and more efficienly.

With these things implemented, most of the tricks I explain in my blog will become redundant and I’ll finally have some more time to spend with the family.

Ever wondered why a condition like this:

WHERE TRUNC(mydate) = TRUNC(SYSDATE)

, which searches for the current day’s records, is so elegant but so slow?

Of course this is because even if you create an index on mydate, this index cannot be used.

The expression in the left part of the equality is not a mydate. The database engine cannot find a way to use an index to search for it. It is said that this expression is not sargable.

Now, a little explanation about the indexes and sargability. If you are familiar with these, you can skip this chapter. But beware that this chapter is the only one illustrated, so skipping it will make the article too boring to read.

Ahem.

To locate a record in a B-Tree index, the keys of the index should be compared to the value being searched for.

Let’s consider this sample B-Tree index:



As you can see, this structure maintains the record order.

Within one page, records are just sorted; the links between the pages obey the sorting order too. A binary search can be used to locate the record on a page; if the search resulted in a pair of adjacent records and the search expression is within the range of these records, then we just follow the link between them. All records behind this link will belong to the range, no matter how deep is the tree.

This works well when we search for the exact field that was indexed.

But what happens when we search for a derived expression?

SELECT  *
FROM    mytable
WHERE   value % 3 = 1

The B-Tree itself of course does not change. But, being casted to the expression we are searching for, the values stored will look like that:

As you can see, there is no order anymore. The records are neither ordered within one page, nor the links follow the order. A 1 we are searching for can appear anywhere in the tree. There is no other way than to traverse the whole tree and compare each record.

It is said that this expression is unsargable: the index can not be used to search for this expression.

But let’s consider another example:

SELECT  *
FROM    mytable
WHERE   value + 3 = 10

Here’s how the B-Tree looks now from the point of view of the expression in the WHERE clause

Now, we have a perfectly valid B-Tree: everything stays in order and the search algorithms can still be used.

Another example (DIV is the integer division operator):

SELECT  *
FROM    mytable
WHERE   value DIV 3 = 3

Here’s the B-Tree as seen from the expression’s side:

The order persists.

This B-Tree is a little bit harder to traverse, since the key values are not unique anymore. But the same values of the keys are still contiguous, and the order between the different keys is still maintained.

Is this case, the algorithm should be changed just a little. When a key is found in the database, we should just continue searching to the left until we find the very first occurrence of the key with lower value. Then we continue searching to the right and return all the records with the correct value of the key.

This problem is of course well known and has long since been solved by all database systems: you can create an index on a non-unique field.

Finally, let’s consider one more expression, a very simple one:

SELECT  *
FROM    mytable
WHERE   20 - value = 10

and how it sees the B-Tree:

In this case, the order is reversed. But there still is the order. The only difference is that the values are sorted right to left, not left to right, and all traversals should be made according to these new directions.

We see that some expressions break the B-Tree order, while the other ones maintain it.

What is the difference between the two?

For an expression to maintain the B-Tree order, it is necessary and sufficient for this expression to be a monotonic function of the argument being indexed.

Here’s the definition of the monotonic function from Wikipedia:

In calculus, a function f defined on a subset of the real numbers with real values is called monotonic (also monotonically increasing, increasing or non-decreasing), if for all x and y such that x ≤ y one has f(x) ≤ f(y), so f preserves the order. Likewise, a function is called monotonically decreasing (also decreasing, or non-increasing) if, whenever x ≤ y, then f(x) ≥ f(y), so it reverses the order.

By definition, the monotonic function preserves (or reverses) the order, but does not break it.

This means that if an index is built over an expression, it also can be used to search over any monotonic function of this expression.

However, most query optimizers do not take this fact into account, while this behavior is relatively easy to implement.

Sample implementation

There will be many new keywords. I will introduce them all at the beginning and explain them later in the article. Also, there will be many fake functions, like NEXT_COLLATION_CHARACTER or EXTRACT(calendar_date), which are absent in the actual engines but their name easily tells their purpose (and it’s quite simple to implement them anyway). These functions are used to demonstrate the concept.

A sargable function, as I shown above, needs to expose some monotony:

function(arg1, …) monotony_declaration

Function of two or more arguments can be sargable by some or all of it’s arguments:

monotony_declaration := MONOTONIC ( monotony_domain | OVER(arg1) monotony_domain [, …])

When the function is a single-argument function, OVER clause should be omitted.

monotony_domain := (monotony | PIECEWISE piecewise_monotony)

Monotony defines the direction and uniqueness of the function on the given range of the function’s domain:

monotony := (STRICTLY INCREASING | INCREASING | STRICTLY DECREASING | DECREASING | UNDEFINED ) [ inversion_clause ]

  • If monotony is STRICTLY INCREASING, it can be used for the B-Tree searches without changing the search order. A UNIQUE set of values is mapped to a UNIQUE set of the results of the function. The cardinality of the function over the indexed expression is the same as that of the expression.
  • If monotony is INCREASING, it can be used for the B-Tree searches without changing the search order. A UNIQUE set of values may be mapped to a non-unique set of the results, and no assumptions should be made about the uniqueness of the results. The cardinality of the function over the indexed expression may be not the same as that of the expression.
  • If monotony is STRICTLY DECREASING, it can be used for the B-Tree searches, changing the search order. A UNIQUE set of values is mapped to a UNIQUE set of the results of the function. The same algorithms that are used for scanning the index in the opposite order should be applied. The cardinality of the function over the indexed expression is the same as that of the expression.
  • If monotony is STRICTLY DECREASING, it can be used for the B-Tree searches, changing the search order. A UNIQUE set of values is mapped to a UNIQUE set of the results of the function. The same algorithms that are used for scanning the index in the opposite order should be applied. The cardinality of the function over the indexed expression may be not the same as that of the expression.
  • There is one more flavor, UNDEFINED, which states that the function is not monotonic or its monotony should not be relied upon. This has no sense as such, but can be used in the complex constructs we shall discuss below.

If the function is monotonic over the whole domain, then monotony is provided right after the keyword MONOTONIC.

However, a function can be piecewise monotonic. Its domain may consist of several (possibly infinite) number of pieces, with the function being monotonic over each piece but not across them.

Piecewise monotony

piecewise_monotony := PIECEWISE (constant_piece_definition | functional_piece_definition)

The pieces can be defined either by a set of constants or by another monotonic function, in which the pieces are defined by distinct values of the piece-defining function.

constant_piece_definition := WHEN VALUE [STRICTLY] LESS THAN const monotony, [ … ], WHEN VALUE LESS THAN MAXVALUE monotony

Each const should be greater than the previous, and the last clause should be WHEN VALUE THEN MAXVALUE. The const comparison can be declared as strict: this may be important for handling discontinuities.

functional_piece_definition := DEFINED BY piece_defining_function(arg) CASE ( (WHEN expression(PIECE) THEN monotony) [ (…) | ELSE monotony ] ) END

Piece-defining function should itself be a monotonic function (but not a strict monotonic) over all domain of the original function.

The point of having this function is to split the domain into several pieces which themselves are sargable (hence the requirement for non-strict monotony), and then define the monotony within each piece.

The monotony for each piece is defined by SQL’s standard CASE WHEN … THEN … ELSE … END control flow construct (which of course returns monotonies instead of values).

Two or more arguments

A function of two or more arguments can be sargable too, as defined by monotony_declaration.

Each OVER clause defines monotony over one of the fuction’s arguments.

If an indexed field is used as a corresponding argument to this function and the other arguments are provided as constants or values from the leading table in the join, then the function is sargable.

Superpositions

When dealing with the superposition of functions, some assumptions can be made:

  • A superposition of any number of STRICTLY MONOTONIC functions is STRICTLY MONOTONIC
  • A superposition of any number of STRICTLY MONOTONIC functions and at least one MONOTONIC function is MONOTONIC (as opposed to STRICTLY MONOTONIC)
  • A superposition of an INCREASING and a DECREASING function is DECREASING
  • A superposition of an INCREASING and an INCREASING functions, or that of a DECREASING function and a DECREASING function is INCREASING

This only concerns functions monotonic over all domain (that is without PIECEWISE expressions).

Inversions

inversion_clause := INVERSE ( inverse_expression | FROM inverse_expression [ EXACT [ EXCLUDE ] ] TO inverse_expression [ EXACT [ EXCLUDE ] ] )

inverse_expression := sql_expression ([ PIECE ], [ RESULT ], [ non-over-arg1 [ ,… ] ])

Each monotony may be provided with an inversion_clause, which maps the results back to the values of the arguments.

These expressions can accept the following arguments:

  • RESULT. This is the result of the function which needs to be mapped back to the value of the argument
  • PIECE. This is the value defined by the piece-defining function. This only makes sense in a function-defined piecewise monotony
  • non-over-arg. This is the argument to the function not used in the OVER clause. This only makes sense for multiple-argument functions.

Inversion clause may come in two forms: scalar inversion and range inversion.

  • A scalar inversion maps the function result to the single value of the argument.

    If the monotony is strict, then the value of the expression is treated as a unique mapping. The decision on whether to return the value or not is made solely on the basis of the inverse result: function is not applied to the values taken from the index for more fine filtering.

    If the monotony is not strict, them the value of the expression is treated as a reference point. Key reads back and forward to this reference point should be made with the function being applied to the result of each key read.

  • A range inversion maps the function result back to the range of values that are guaranteed to yield the RESULT, being provided as an argument to the function. The first expression provides the start value of the range, the second expression provides the end value.

    Within the range, the results are returned from the index without any additional checking.

    If any of the range expressions is marked as EXACT, then no values are returned outside the range in the corresponding direction. Otherwise, key reads are made in this direction and the function is applied to the results of the key reads to do the fine checking.

    With EXACT clause, it is also possible to provide EXCLUDE clause. Depending on its presence, the corresponding range will be treated as including or not including the boundary.

The point of allowing inexact inverse functions is to deal with the rounding errors introduced by floating point operations. Even if in theory the function can uniquely map back to a single value or a range of values, in practice, this may be not the case.

The B-Tree search is possible even without providing inverse functions at all. However, if inverse functions are provided, the number of the function evaluations is kept to the minimum: the function is only evaluated for several keys outside of the guaranteed range.

Examples

  • EXP(arg FLOAT) MONOTONIC INCREASING INVERSE LN(RESULT)

    Exponent function is increasing over the whole set of natural numbers. Its inverse function is natural logarithm.

    Due to possible rounding errors, this function cannot be declared as strict.

  • YEAR(arg DATETIME) MONOTONIC INCREASING
    INVERSE
    FROM DATE_TRUNC('year', RESULT) EXACT
    TO DATE_TRUNC('year', RESULT) + INTERVAL 1 YEAR EXACT EXCLUDE

    YEAR is an increasing function but not a strictly increasing one: two distinct dates can share the same year. That’s why we provide two inverse functions: the first one provides the minimal possible date for a given year, the second one provides the maximal possible one. The latter function is marked as EXCLUDE, since the first date of the next year does not belong to the current year’s domain piece, but any value strictly less than it (and greater than the year’s beginning date) does.

  • ABS(arg INT) MONOTONIC PIECEWISE ON
    VALUES STRICTLY LESS THAN 0 STRICTLY DECREASING INVERSE -RESULT,
    VALUES LESS THAN MAXVALUE STRICTLY INCREASING INVERSE RESULT

    The absolute value function is piecewise monotonic: it is strictly decreasing for the values below zero and strictly increasing for the values of zero and above.

    The inverse functions for each piece are the negation of the result and result itself, accordingly.

  • FLOOR(arg FLOAT) MONOTONIC INCREASING
    INVERSE
    FROM RESULT EXACT
    TO RESULT + 1 EXACT EXCLUDE

    The nearest lower integer function is monotonically increasing, almost the same as YEAR described above.

  • OPERATOR_DIVISION(arg1 FLOAT, arg2 FLOAT)
    MONOTONIC OVER (arg1)
    CASE WHEN arg2 > 0 THEN INCREASING INVERSE RESULT * arg2
    WHEN arg2 = 0 THEN UNDEFINED
    WHEN arg2 < 0 THEN DECREASING INVERSE RESULT * arg2
    END

    This function defines division operator.

    It is monotonic over the first argument. This means that the second argument should be provided as the run-time constant for it to be sargable: no field from the table being searched for can be used as a second argument to this function.

    This function is monotonic over the whole domain of arg1, but the monotony depends on the value of arg2.

    In theory, this function is strictly monotonic, but in practice, rounding errors can make the function yield same results for different values of arg1. That's why this function should be declared non-strict and a single inverse expression should be provided.

  • OPERATOR_PLUS(arg1 FLOAT, arg2 FLOAT) MONOTONIC
    OVER (arg1) STRICTLY INCREASING INVERSE RESULT - arg2,
    OVER (arg2) STRICTLY INCREASING INVERSE RESULT - arg1

    This function defines the + operator.

    This is quite simple: the function is strictly increasing over both arguments, and the inverse function is just a subtraction.

  • MONTH(arg DATETIME) MONOTONIC PIECEWISE
    DEFINED BY DATE_TRUNC('year', arg)
    INCREASING
    INVERSE
    FROM PIECE + INTERVAL RESULT - 1 MONTH EXACT
    TO PIECE + INTERVAL RESULT MONTH EXACT EXCLUDE

    MONTH returns the month number within a year.

    This function is piecewise monotonic: within a year, higher dates always return same of higher months. Whenever the year changes, monotony breaks. DATE_TRUNC is monotonic and not strict, and, hence, can be used to define the pieces.

    The reverse functions use both PIECE and RESULT arguments. The PIECE argument defines the current year, the RESULT is used to find the beginning of the current and the next months.

  • COS(arg FLOAT) MONOTONIC PIECEWISE
    DEFINED BY FLOOR(arg / PI())
    CASE PIECE % 2
    WHEN 0 THEN DECREASING INVERSE PIECE * PI() + ACOS(RESULT)
    ELSE INCREASING INVERSE (PIECE + 1) * PI() - ACOS(RESULT)
    END

    Cosine is piecewise monotonic too. The pieces are defined by the cosine half-waves. Depending on the piece, it can be increasing or decreasing. The half-wave number is defined by FLOOR(arg / PI()). This piece-defining function, being a superposition of increasing functions, is increasing too.

    On the even half-waves, the function is decreasing; on the odd half-waves, it is increasing. In principle, the function monotony is strict, but due to the possible rounding errors, it is marked as non-strict and only one inverse function provided.

  • SIN(arg FLOAT) MONOTONIC PIECEWISE
    DEFINED BY FLOOR(arg / PI() - 0.5)
    CASE PIECE % 2
    WHEN 0 THEN DECREASING INVERSE PIECE * PI() + ASIN(RESULT)
    ELSE INCREASING INVERSE PIECE * PI() - ASIN(RESULT)
    END

    Same as cosine above, with a π / 2 shift.

  • LEFT(arg VARCHAR, length INT) MONOTONIC
    OVER(arg) INCREASING
    INVERSE
    FROM LEFT(arg, length) EXACT
    TO LEFT(arg, length - 1) || NEXT_COLLATION_CHARACTER(SUBSTRING(arg, length, 1)) EXACT EXCLUDE

    Function LEFT takes the leading characters from the string.

    It is sargable over the first argument: in the string is indexed, you can always search for a leading substring. I had to make up the function NEXT_COLLATION_CHARACTER, which would take next possible alphabet character according to the current collation.

    Note that SQL Server and MySQL have traces of such optimization. In both systems, LIKE predicate is sargable if the search string does not contain the leading wildcards.

    If you build a plan for a predicate like column LIKE 'test%', you will see that is will use the index seek in this range:

    column >= 'test' AND column < 'tesU'

    This behavior is exactly like described above.

Search algorithm

To select the values matching this condition:

function(col1 [ , arg2, … ]) = const

, the following procedures should be made:

  1. If the function is not sargable against the first argument, then just do the full table or index scan.

  2. If the function is sargable against the first argument, check its monotony.

  3. If the function is monotonic over all domain:

    1. If no inversion function is defined, then locate the values in the B-Tree using the B-Tree search algorithm and applying the function to each key before the comparison. The monotony direction (increasing or declreasing) should of course be taken into account.

    2. If the function is strictly monotonic and a single inverse function is defined, apply the inverse function to the value of const, find the corresponding value of col1 and locate it in the index using B-Tree search algorithm.

    3. If the function is monotonic, but not strictly, check the inversion type:

      1. If a scalar inverse expression is defined, then:

        1. Apply the inverse function to const and find the key values in the B-Tree closest to the value of inverse_expression(const)

        2. Starting from inverse_expression(const), iterate through the index back and forward, applying function to the each key found until the function exceeds (or falls short of) const, returning the records found.

      2. If a range inverse expression is defined, then:

        1. Find the beginning and the ending values of the col1 range that yields const
          If the beginning of range is not exact, iterate keys backwards, applying the function to each key found until the function equals to const.

        2. Iterate the keys forward, returning the records found, until the end of the range is reached.

        3. If the range end is not exact, then iterate keys beyond the range, applying the function to the keys found, until the function value exceeds or falls short of const

  4. If the function is piecewise monotonic:

    1. If the pieces are defined by a set of constants, split the domain into the pieces, then apply the steps above to each piece separately with respect to the monotony of each piece.

    2. If the pieces are defined by a function:

      1. Use loose index scan to build a distinct list of the function's values (each defining a piece). The loose index scan will jump over the B-Tree, starting from the minimal (or maximal) value of the function and recursively searching for the greater (or the lesser value). Each distinct value will reveal the least and the greatest value of the col1 belonging to its piece.

      2. Within the range defined by these values, apply the steps above, with respect to the monotony of each piece.

    3. If for any of the pieces the monotony is UNDEFINED, then just apply the function to every key in the piece's range, filtering the results.

User-defined functions

All examples above were applied to the built-in functions. In fact, they don't need declaration and all clauses above were added only to demonstrate the point.

However, with a little effort, the user-defined functions can be made sargable too.

Here are some suggestions on how to make it:

  1. To be sargable, a user-defined function should be strictly deterministic, that is its result should depend on and only on the values of the arguments (specifically, it should not depend on the database state)
  2. If a function is a superposition of the monotonic functions (with the additional constraints stated above), it should be automatically treated as monotonic.

  3. If a function is declared as monotonic by a user, special algorithms should be used to check its monotony (and hence sargability):

    1. Sargability of a monotonic user-defined function should be a property of index-function pair. Function's declared monotony is a necessary but not sufficient condition for sargability.

      By default, sargability of index-function pair is unknown.

    2. Whenever a query is issued which could be satisfied by using the index to search for the function values, the index-function pair is marked as potentially sargable. This property can be made dynamic: say, 10 or 100 queries should be issued before the sargability of the index-function pair can be checked.

    3. When the query threshold is reached (this means that the certain number of queries which possibly could use the index had been run), the index checking process is initiated.
    4. The database job which is responsible for collecting the index statistics, runs over the index and checks the declared monotony of the function over the given argument: within each piece, higher values of argument should map to higher (or lower, or not higher, or not lower) values of the function. If the function is piecewise monotonic with pieces defined by a user-defined function as well, the monotony of the piece-defined function should also be checked in the same routine.
    5. If the check succeeds, the index-function is marked as sargable both for the function and its piece-defining function (if any).
    6. At the same time, the statistics routine should collect the additional statistics for the function as if it was the index over the function's values.

    7. If an index is marked as usable for any given function, all DML operations against this index should perform an additional validation for the function monotony:

      1. Whenever a new record is inserted into the index, the function should be applied to the values of the key inserted and both of its neighbors. The value of the function over the key inserted should be between (or, depending on the function's monotony, strictly between) the values of the function over the neighboring keys.
      2. If the function is declared as piecewise monotonic with a piece defining function, the latter is checked first.

      3. If any of the neighboring keys belongs to a piece different to that of the key being inserted, that neighboring key will not participate in the check.

    8. A failure to prove the function's monotony invalidates the function's usage in all indexes and also invalidates the sargability of all functions that use it as a piece-defining function. Additionally, the values of the arguments which are proven to violate the function's monotony are recorded with the function's metadata.
    9. Any change to the function's definition invalidates all sargability information linked with this function, as well as linked with the functions that uses it to define the pieces.

    10. For any index-function pair, the validation and statistics collection process can be initiated manually. The database engine should provide means for that.

Points of interest

Function sargability may be used to improve many queries, specifically:

  • Searching for dates within a given period (month, date, year) without a need to calculate the ranges manually:

    SELECT  *
FROM    mytable
WHERE   EXTRACT(YEAR_MONTH FROM col1) = '201002'

  • Searching for birthdates and calendar events:

    SELECT  *
FROM    mytable
WHERE   EXTRACT(calendar_date FROM mydate) = '0310'

  • Searching for the substrings:

    SELECT  *
FROM    mytable
WHERE   LEFT(value, 10) = 'abcdefghij'

  • Searching for the COALESCE'd values:

    SELECT  *
FROM    mytable
WHERE   COALESCE(value, 'test') = 'test'

and many others.

I really hope that this will be added to the major database engines in the nearest future.

]]> http://explainextended.com/2010/02/19/things-sql-needs-sargability-of-monotonic-functions/feed/ 3 Tags in nested sets: efficient indexing http://explainextended.com/2010/02/16/tags-in-nested-sets-efficient-indexing/ http://explainextended.com/2010/02/16/tags-in-nested-sets-efficient-indexing/#comments Tue, 16 Feb 2010 20:00:11 +0000 Quassnoi http://explainextended.com/?p=4326 Answering questions asked on the site.

Travis asks:

I just read your excellent post: Hierarchical data in MySQL: parents and children in one query.

I am currently trying to do something very similar to this. The main difference is, I am working with data that is in a nested set.

I would like to construct a query that will return a resultset of nodes matched with LIKE, the ancestors of each of these nodes, and the immediate children of each of these nodes.

This is a very interesting question: it allows to demonstrate the usage of three types of indexes usable by MyISAM tables: BTREE, SPATIAL and FULLTEXT.

Nested sets is one of two models that are most often used to store hierarchical data in a relational table (the other one being adjacency list model). Unlike adjacency list, nested sets does not require ability to write recursive queries, which SQL originally lacked and still lacks now in MySQL (albeit it can be emulated to some extent). It is widely used in MySQL world.

I described both methods and their comparison in the article I wrote some time ago:

The main problem with the nested sets model is that though it is extremely fast with selecting descendants of a given node, it is very slow in selecting ancestors.

This is because of the way the B-Tree indexes work. They can be used to query for values of a column within the range of two constants, but not for the values of two columns holding a single constant between them. One needs the first condition to select the children (found between the lft and rgt of the given node), and the second condition to select the ancestor (with lft and rgt containing the lft and rgt of the given node).

That’s why selecting the children is fast and selecting the ancestors is slow.

To work around this, the sets that form the hierarchy can be described as geometrical objects, with the larger sets containing the smaller sets. These sets can be indexed with a SPATIAL index which is designed specially for this purpose and both children and ancestors can be queried for very efficiently.

Unfortunately, finding the depth level is quite a hard task for the nested sets model even with the SPATIAL indexes.

It would be quite an easy task is MySQL supported recursion: we could just run a query to find the siblings of each record by skipping their whole domains recursively.

However, MySQL’s recursion support is very limited and it relies on the session variables, which are not recommended to use in the complex queries.

To cope with this, we need to mix the nested sets and the adjacency list models. Hierarchy will be stored in two seemingly redundant ways: the unique parent and the LineString representing the nested sets.

This will help us to use the R-Tree index to find all ancestors of a given node and also use B-Tree index to find its immediate children.

Finally, the question mentions using LIKE to find the initial nodes. LIKE predicate with the leading wildcards is not sargable in MySQL. However, it seems that the leading wildcards are only used to split the words. In this case, a FULLTEXT index and the MATCH query would be much more efficient, since FULLTEXT index allows indexing a single record with several keys (each one corresponding to a single word in the column’s text), so a search for the word in the space separated or a comma separated list uses the index and is much faster than scanning the whole table.

Hence, the query would use all three main types of indexes: BTREE, SPATIAL and FULLTEXT.

To illustrate everything said above, let’s create a sample table:

Table creation details

This table represents a 7 level hierarchy, with 5 children to each non-leaf item and combines nested sets and adjacency list models.

The sets are stored in the plain lft and rgt columns as well as in the combined column of type LineString which represents a diagonal of a box horizontally spanning the interval from lft to rgt.

Selecting nodes using FULLTEXT

First, let’s select the nodes tagged with a given tag.

RLIKE can be used for that but it is not very efficient:

SELECT  id, tags
FROM    t_hierarchy
WHERE   tags RLIKE '[[:<:]]tag13480[[:>:]]'

View query results

This query uses full table scan and runs for 1.25 second.

To improve the query, we should rewrite the WHERE condition using MATCH predicate (which in its turn allows a FULLTEXT index to be used for the search):

SELECT  id, tags
FROM    t_hierarchy
WHERE   MATCH(tags) AGAINST('+tag13480' IN BOOLEAN MODE)
id tags
3 tag13480 tag33087 tag124996
440166 tag248489 tag271789 tag13480
212847 tag104605 tag13480 tag53585
161378 tag181040 tag231320 tag13480
324394 tag13480 tag181947 tag269297
400074 tag13480 tag176642 tag242772
6 rows fetched in 0.0005s (0.0043s)
id select_type table type possible_keys key key_len ref rows filtered Extra
1 SIMPLE t_hierarchy fulltext fx_hierarchy_tags fx_hierarchy_tags 0 1 100.00 Using where 
select `20100216_index`.`t_hierarchy`.`id` AS `id`,`20100216_index`.`t_hierarchy`.`tags` AS `tags` from `20100216_index`.`t_hierarchy` where (match `20100216_index`.`t_hierarchy`.`tags` against ('+tag13480' in boolean mode))

This query returns the same results but does it much faster, in only 4 ms.

Selecting ancestors using SPATIAL

Now, when we have a list of nodes, we should build the list of ancestors for each node.

Since our model combines adjacency list and nested sets, it is possible to use either representation to build a query. However, the adjacency list model requires recursion, and, while it is possible to emulate it, it only works for a single-node query.

With nested sets, selecting the list of ancestors is much more simple: we should just selecting all records whose sets contains the sets of the node. This can be done using MBRContains (which is capable of using the SPATIAL index).

The query, however, will return us the ancestors in a plain list. To find out the level of each ancestor, we should put some more effort. Since the sets are nested and the lft and rgt fields naturally maintain the hierarchical order, it is enough just to enumerate the ancestors in that order. It would be very simple to do — if only MySQL supported ROW_NUMBER(). But it doesn’t, of course, so to enumerate the ancestors we should self-join them and just count the number of each ancestor’s ancestors:

SELECT  hc.id, ha.id, COUNT(*) AS cnt
FROM    t_hierarchy hc
STRAIGHT_JOIN
        t_hierarchy ha
ON      MBRContains(ha.sets, hc.sets)
STRAIGHT_JOIN
        t_hierarchy hcnt
ON      MBRContains(hcnt.sets, ha.sets)
WHERE   MATCH(hc.tags) AGAINST('+tag13480' IN BOOLEAN MODE)
GROUP BY
        hc.id, ha.id
ORDER BY
        hc.id, cnt

View query results

The algorithm that calculates the level of each ancestor is not very efficient, however, it does its job quite well, and the query completes in only 20 ms.

Selecting immediate children using BTREE

Selecting the immediate children is the easiest task: we just need an equijoin on parent. The level of each of the node’s children will be that of the node plus 1, so calculating it is quite simple too:

SELECT  h.id, hc.id, cnt + 1
FROM    (
        SELECT  hn.id, COUNT(*) AS cnt
        FROM    t_hierarchy hn
        STRAIGHT_JOIN
                t_hierarchy hcnt
        ON      MBRContains(hcnt.sets, hn.sets)
        WHERE   MATCH(hn.tags) AGAINST('+tag13480' IN BOOLEAN MODE)
        GROUP BY
                hn.id
        ) h
JOIN    t_hierarchy hc
ON      hc.parent = h.id

View query results

Putting it together

Now we should just combine the two queries and apply nice formatting to them:

SELECT  h.id,
        CONCAT(LPAD('', (level - 1) * 2, ' '), h.data) AS name,
        CASE WHEN hq.id = hq.node THEN '*' ELSE '' END AS hit
FROM    (
        SELECT  hc.id AS node, ha.id AS id, COUNT(*) AS level
        FROM    t_hierarchy hc
        STRAIGHT_JOIN
                t_hierarchy ha
        ON      MBRContains(ha.sets, hc.sets)
        STRAIGHT_JOIN
                t_hierarchy hcnt
        ON      MBRContains(hcnt.sets, ha.sets)
        WHERE   MATCH(hc.tags) AGAINST('+tag13480' IN BOOLEAN MODE)
        GROUP BY
                hc.id, ha.id
        UNION ALL
        SELECT  h.id, hc.id, cnt + 1 AS level
        FROM    (
                SELECT  hn.id, COUNT(*) AS cnt
                FROM    t_hierarchy hn
                STRAIGHT_JOIN
                        t_hierarchy hcnt
                ON      MBRContains(hcnt.sets, hn.sets)
                WHERE   MATCH(hn.tags) AGAINST('+tag13480' IN BOOLEAN MODE)
                GROUP BY
                        hn.id
                ) h
        JOIN    t_hierarchy hc
        ON      hc.parent = h.id
        ) hq
JOIN    t_hierarchy h
ON      h.id = hq.id
ORDER BY
        node, level, lft
id name hit
3 Value 3 *
16 Value 16
17 Value 17
18 Value 18
19 Value 19
20 Value 20
1 Value 1
10 Value 10
51 Value 51
257 Value 257
1290 Value 1290
6454 Value 6454
32275 Value 32275
161378 Value 161378 *
2 Value 2
13 Value 13
67 Value 67
340 Value 340
1702 Value 1702
8513 Value 8513
42569 Value 42569
212847 Value 212847 *
3 Value 3
20 Value 20
103 Value 103
518 Value 518
2594 Value 2594
12975 Value 12975
64878 Value 64878
324394 Value 324394 *
4 Value 4
25 Value 25
127 Value 127
639 Value 639
3200 Value 3200
16002 Value 16002
80014 Value 80014
400074 Value 400074 *
5 Value 5
27 Value 27
140 Value 140
704 Value 704
3521 Value 3521
17606 Value 17606
88033 Value 88033
440166 Value 440166 *
46 rows fetched in 0.0039s (0.0466s)
id select_type table type possible_keys key key_len ref rows filtered Extra
1 PRIMARY <derived2> ALL 46 100.00 Using temporary; Using filesort
1 PRIMARY h eq_ref PRIMARY PRIMARY 4 hq.id 1 100.00
2 DERIVED hc fulltext sx_hierarchy_sets,fx_hierarchy_tags fx_hierarchy_tags 0 1 100.00 Using where; Using temporary; Using filesort
2 DERIVED ha range sx_hierarchy_sets sx_hierarchy_sets 34 1 48828000.00 Range checked for each record (index map: 0×10)
2 DERIVED hcnt range sx_hierarchy_sets sx_hierarchy_sets 34 1 48828000.00 Range checked for each record (index map: 0×10)
3 UNION <derived4> ALL 6 100.00
3 UNION hc ref ix_hierarchy_parent ix_hierarchy_parent 4 h.id 9207 100.01
4 DERIVED hn fulltext sx_hierarchy_sets,fx_hierarchy_tags fx_hierarchy_tags 0 1 100.00 Using where; Using temporary; Using filesort
4 DERIVED hcnt range sx_hierarchy_sets sx_hierarchy_sets 34 1 48828000.00 Range checked for each record (index map: 0×10)
UNION RESULT <union2,3> ALL 
select `20100216_index`.`h`.`id` AS `id`,concat(convert(lpad('',((`hq`.`level` - 1) * 2),' ') using utf8),`20100216_index`.`h`.`data`) AS `name`,(case when (`hq`.`id` = `hq`.`node`) then '*' else '' end) AS `hit` from (select `20100216_index`.`hc`.`id` AS `node`,`20100216_index`.`ha`.`id` AS `id`,count(0) AS `level` from `20100216_index`.`t_hierarchy` `hc` straight_join `20100216_index`.`t_hierarchy` `ha` straight_join `20100216_index`.`t_hierarchy` `hcnt` where (match `20100216_index`.`hc`.`tags` against ('+tag13480' in boolean mode)) group by `20100216_index`.`hc`.`id`,`20100216_index`.`ha`.`id` union all select `h`.`id` AS `id`,`20100216_index`.`hc`.`id` AS `id`,(`h`.`cnt` + 1) AS `level` from (select `20100216_index`.`hn`.`id` AS `id`,count(0) AS `cnt` from `20100216_index`.`t_hierarchy` `hn` straight_join `20100216_index`.`t_hierarchy` `hcnt` where ((match `20100216_index`.`hn`.`tags` against ('+tag13480' in boolean mode)) and contains(`20100216_index`.`hcnt`.`sets`,`20100216_index`.`hn`.`sets`)) group by `20100216_index`.`hn`.`id`) `h` join `20100216_index`.`t_hierarchy` `hc`) `hq` join `20100216_index`.`t_hierarchy` `h` where (`20100216_index`.`h`.`id` = `hq`.`id`) order by `hq`.`node`,`hq`.`level`,`20100216_index`.`h`.`lft`

For each matching node, the query returns the node itself, its ancestors and its immediate children. The matching nodes are marked with the asterisk in the field hit.

The query efficiently combines the FULLTEXT, SPATIAL and BTREE indexes and completes in only 40 ms.

Hope that helps.

I’m always glad to answer the questions regarding database queries.

Ask me a question

]]> http://explainextended.com/2010/02/16/tags-in-nested-sets-efficient-indexing/feed/ 0