EXPLAIN EXTENDED » PostgreSQL http://explainextended.com How to create fast database queries Wed, 25 Aug 2010 13:29:38 +0000 en hourly 1 http://wordpress.org/?v=3.0.1 Date ranges: overlapping with priority http://explainextended.com/2010/04/07/date-ranges-overlapping-with-priority/ http://explainextended.com/2010/04/07/date-ranges-overlapping-with-priority/#comments Wed, 07 Apr 2010 19:00:31 +0000 Quassnoi http://explainextended.com/?p=4663 Answering questions asked on the site.

Jason Foster asks:

We have a table of student registrations:

Students
student_code course_code course_section session_cd
987654321 ESC102H1 Y 20085
998766543 ELEE203H F 20085

course_code and course_section identify a course, session_cd is an academic session, e. g. 20085, 20091, 20079.

The courses (stored in another table) have associated values for engineering design, complementary studies, etc., like that:

Courses
course_code course_section start_session end_session design science studies
ESC102H1 F 20071 20099 10 0 0
AER201Y1 Y 20059 NULL 0 0 30

, or like that:

In-house courses
course_code course_section student_code design science studies
ESC102H1 F 998766543 10 0 0

We are required by an external accreditation body to add up all of the units of engineering design, complementary studies, etc., taken by an individual student.

Where it gets really messy is that we have multiple data feeds for the associated values of courses. For example we have a set from the Registrar’s Office, the Civil Department, our In-House version, etc.

The rule is that In-House beats Civil beats the Registrar’s Office in the case of any duplication within the overlapping intervals.

The session_cd is of the form YYYY{1,5,9}.

Basically, we have three sets here.

To get the course hours for a given student we should find a record for him in the in-house set, or, failing that, find if the session is within the ranges of one of the external sets (Civil or Registrar). If both ranges contain the academic session the student took the course, Civil should be taken.

The first part is quite simple: we just LEFT JOIN students with the in-house courses and get the hours for the courses which are filled. The real problem is the next part: searching for the ranges containing a given value.

As I already mentioned in the previous posts, relational databases are not in general that efficient for the queries like that. It’s easy to use an index to find a value of a column within a given range, but B-Tree indexes are of little help in searching for a range of two columns containing a given value.

However, in this case, the data domain of session_cd is quite a limited set. For each pair of session_start and session_end it is easy to create a set of all possible values between session_start and session_end.

The overlapping parts of the session ranges from the two sets will yields two records for each of the sessions belonging to the range. Of these two records we will need to take the relevant one (that is Civil) by using DISTINCT ON with the additional sorting on the source (Civil goes first).

Then we just join the relevant records to the subset of the students which does not have corresponding records in the in-house version.

Finally, we need to union this with the in-house recordset.

Pictures

Here’s the same thing in pictures:

  1. Within each source, courses are defined by the a single record holding the start and end session:

  2. To find the courses superposition (regarding the priority) we split each range into a number of records, each corresponding to a single session, then combine these records in a singe recordset, ordered by session then by source:

  3. From each session, we take a single record with the higher priority and use to to join with the students table:

Query

Now, let’s create some sample tables and see how it works:

Table creation details

The tables contain 500 students, random civil and registrar ranges for 200 courses, and 20,000 in-house records for first 100 students.

And here’s the query (limited to return first 10 records for the sake of readability):

SELECT  1 AS sourse, s.id AS student, i.course AS course, hours1, hours2, hours3
FROM    t_student s
JOIN    t_inhouse i
ON      i.student = s.id
        AND i.course = s.course
UNION ALL
SELECT  source, s.id, q.id, hours1, hours2, hours3
FROM    t_student s
JOIN    (
        SELECT  DISTINCT ON (current_session, id) *
        FROM    (
                SELECT  *,
                        ((cs / 3)) * 100 + (ARRAY[1, 5, 9])[cs % 3 + 1] AS current_session
                FROM    (
                        SELECT  *,
                                generate_series(e, l) AS cs
                        FROM    (
                                SELECT  *,
                                        session_start * 3 / 100 + CASE (session_start % 100) WHEN 1 THEN 0 WHEN 5 THEN 1 ELSE 2 END AS e,
                                        session_end * 3 / 100 + CASE (session_end % 100) WHEN 1 THEN 0 WHEN 5 THEN 1 ELSE 2 END AS l
                                FROM    (
                                        SELECT  2 AS source, *
                                        FROM    t_civil
                                        UNION ALL
                                        SELECT  3 AS source, *
                                        FROM    t_registrar
                                        ) q
                                ) q
                        ) q
                ) q
        ORDER BY
                current_session, id, source
        ) q
ON      q.current_session = s.session
        AND q.id = s.course
WHERE   NOT EXISTS
        (
        SELECT  NULL
        FROM    t_inhouse ih
        WHERE   ih.student = s.id
                AND ih.course = s.course
        )
ORDER BY
        student, course
LIMIT 10
sourse student course hours1 hours2 hours3
1 1 4 21 25 44
1 1 10 18 49 49
1 1 26 12 26 2
1 1 27 32 38 22
1 1 39 39 27 36
1 1 44 32 26 30
1 1 51 3 21 3
1 1 54 7 10 34
1 1 57 46 34 10
1 1 63 50 18 7
10 rows fetched in 0.0008s (0.0968s) 
Limit  (cost=2323.47..2323.50 rows=10 width=20)
  ->  Sort  (cost=2323.47..2328.52 rows=2019 width=20)
        Sort Key: s.id, i.course
        ->  Append  (cost=817.31..2279.84 rows=2019 width=20)
              ->  Merge Join  (cost=817.31..1837.75 rows=1979 width=20)
                    Merge Cond: ((i.course = s.course) AND (i.student = s.id))
                    ->  Index Scan using t_inhouse_pkey on t_inhouse i  (cost=0.00..825.94 rows=20000 width=20)
                    ->  Sort  (cost=817.21..842.18 rows=9986 width=8)
                          Sort Key: s.course, s.id
                          ->  Seq Scan on t_student s  (cost=0.00..153.86 rows=9986 width=8)
              ->  Nested Loop Anti Join  (cost=362.94..421.91 rows=40 width=24)
                    ->  Hash Join  (cost=362.94..404.44 rows=50 width=28)
                          Hash Cond: ((((((q.cs / 3) * 100) + ('{1,5,9}'::integer[])[((q.cs % 3) + 1)])) = s.session) AND (q.id = s.course))
                          ->  Unique  (cost=59.29..62.29 rows=200 width=40)
                                ->  Sort  (cost=59.29..60.29 rows=400 width=40)
                                      Sort Key: ((((q.cs / 3) * 100) + ('{1,5,9}'::integer[])[((q.cs % 3) + 1)])), q.id, q.source
                                      ->  Subquery Scan q  (cost=0.00..42.00 rows=400 width=40)
                                            ->  Result  (cost=0.00..33.00 rows=400 width=56)
                                                  ->  Append  (cost=0.00..8.00 rows=400 width=56)
                                                        ->  Seq Scan on t_civil  (cost=0.00..4.00 rows=200 width=56)
                                                        ->  Seq Scan on t_registrar  (cost=0.00..4.00 rows=200 width=56)
                          ->  Hash  (cost=153.86..153.86 rows=9986 width=12)
                                ->  Seq Scan on t_student s  (cost=0.00..153.86 rows=9986 width=12)
                    ->  Index Scan using t_inhouse_pkey on t_inhouse ih  (cost=0.00..0.35 rows=1 width=8)
                          Index Cond: ((ih.course = s.course) AND (ih.student = s.id))

The query returns the required hours and the source of these hours for each of the courses a student attended.

Hope that helps.

I’m always glad to answer the questions regarding database queries.

Ask me a question

]]> http://explainextended.com/2010/04/07/date-ranges-overlapping-with-priority/feed/ 0 PostgreSQL: using recursive functions in nested sets http://explainextended.com/2010/03/02/postgresql-using-recursive-functions-in-nested-sets/ http://explainextended.com/2010/03/02/postgresql-using-recursive-functions-in-nested-sets/#comments Tue, 02 Mar 2010 20:00:45 +0000 Quassnoi http://explainextended.com/?p=4528 In the previous article, I discussed a way to improve nested sets model in PostgreSQL.

The approach shown in the article used an analytical function to filter all immediate children of a node in a recursive CTE.

This allowed us to filter a node’s children on the level more efficiently than R-Tree or B-Tree approaches do (since they rely on COUNT(*)).

That solution was pure SQL and it was quite fast, but not optimal.

The drawback of that solution is that it still needs to fetch all children of a node to apply the analytic function to them. This can take much time for the top of the hierarchy. And since the top of the hierarchy is what is what usually shown at the start page, it would be very nice to improve this query yet a little more.

We can do it by creating and using a simple recursive SQL function. This function does not even require PL/pgSQL to be enabled.

Let’s create a sample table:

Table creation details

If we run the query introduced in the previous article to fetch all children up to level 2 from a really top node, we get the following results:

WITH    RECURSIVE
        q AS
        (
        SELECT  id, lft, rgt, 1 AS lvl
        FROM    t_hierarchy
        WHERE   id = 1
        UNION ALL
        SELECT  *
        FROM    (
                SELECT  DISTINCT ON (MAX(hc.rgt) OVER (PARTITION BY q.id ORDER BY hc.lft)) hc.id, hc.lft, hc.rgt, lvl + 1
                FROM    q
                JOIN    t_hierarchy hc
                ON      hc.lft > q.lft
                        AND hc.lft < q.rgt
                WHERE   lvl <= 2
                ORDER BY
                        MAX(hc.rgt) OVER (PARTITION BY q.id ORDER BY hc.lft), hc.lft
                ) q2
        )
SELECT  *
FROM    q

View query details

This runs for almost 15 seconds: too much.

This can be improved by exploiting these two properties of the nested sets model:

  1. The first immediate child of a node is the node holding the first lft next to the node’s lft

  2. The next sibling of a node is the node holding the first lft next to the node’s rgt

If we recursively traverse through the nodes, we can find the first child as well as all of its siblings. This is enough to build a hierarchy, and level filter can be implemented merely by limiting the recursion depth.

However, recursive CTE‘s only allow one recursion level. We cannot nest the WITH clause.

To work around that, we can use PostgreSQL‘s ability to run set-returning functions recursively. We will use the function-based recursion to iterate the parent-child axis, and the CTE-based recursion to iterate siblings axis.

We need to create a function that would take a node’s id on input and return a set of its children on output, with the function recursively applied to each of the children. To find a set of children, we will implement a recursive CTE that finds the first child in the anchor part and the next sibling in the recursive part.

Here’s the function:

CREATE OR REPLACE FUNCTION fn_get_children(id INT, level INT)
RETURNS SETOF INT[] AS
$$
        WITH    RECURSIVE q AS
                (
                SELECT  (hc).id, (hc).lft, (hc).rgt, prgt
                FROM    (
                        SELECT  (
                                SELECT  hc
                                FROM    t_hierarchy hc
                                WHERE   hc.lft > hp.lft
                                        AND hc.lft < hp.rgt
                                ORDER BY
                                        hc.lft
                                LIMIT 1
                                ) hc,
                                rgt AS prgt
                        FROM    t_hierarchy hp
                        WHERE   hp.id = $1
                        ) q2
                UNION ALL
                SELECT  (hc).id, (hc).lft, (hc).rgt, prgt
                FROM    (
                        SELECT  (
                                SELECT  hc
                                FROM    t_hierarchy hc
                                WHERE   hc.lft > q.rgt
                                        AND hc.lft < q.prgt
                                ORDER BY
                                        hc.lft
                                LIMIT 1
                                ) hc,
                                prgt
                        FROM    q
                        WHERE   q.lft IS NOT NULL
                        ) q2
                )
        SELECT  CASE which
                WHEN 1 THEN ARRAY[q.id, $2]
                ELSE fn_get_children(q.id, $2 - 1)
                END
        FROM    (
                VALUES (1), (2)
                ) vals(which)
        CROSS JOIN
                q
        WHERE   q.id IS NOT NULL
                AND $2 > 0
        ORDER BY
                id, which;
$$ LANGUAGE sql;

The function accepts a node’s id and a level on input, and returns a set of arrays, each corresponding to one of the node’s children and its level. The level returned by the function decreases and in fact represents not the level as such, but the number of levels left to reach the filter-set bottom. But since the initial level is user-set, it is easy to cast it to the actual level.

Let’s run the function:

SELECT  c[1], 3 - c[2]
FROM    fn_get_children(1, 2) c;
c ?column?
6 1
34 2
35 2
33 2
31 2
32 2
7 1
38 2
40 2
39 2
36 2
37 2
8 1
41 2
42 2
43 2
44 2
45 2
9 1
47 2
50 2
46 2
48 2
49 2
10 1
53 2
55 2
51 2
52 2
54 2
30 rows fetched in 0.0023s (0.0658s) 
Function Scan on fn_get_children c  (cost=0.00..262.50 rows=1000 width=32)

As we can see, the function returned all children and grandchildren of the node 1 along with their level, and did it in only 65 ms.

]]> http://explainextended.com/2010/03/02/postgresql-using-recursive-functions-in-nested-sets/feed/ 0 PostgreSQL: nested sets and R-Tree http://explainextended.com/2010/03/01/postgresql-nested-sets-and-r-tree/ http://explainextended.com/2010/03/01/postgresql-nested-sets-and-r-tree/#comments Mon, 01 Mar 2010 20:00:04 +0000 Quassnoi http://explainextended.com/?p=4506 A feedback on one of my previous articles comparing adjacency list and nested sets models for PostgreSQL.

Jay writes:

In your series on adjacency lists vs nested sets, you discuss geometric types and R-Tree indexes in MySQL, but you don’t discuss them when discussing the same subject with PostgreSQL, which also has geometric types and R-Tree indexing (mostly available through GiST indexes).

To make it simple I added the following line after the data insertion part of the script at Nested Sets – Postgresql:

ALTER TABLE t_hierarchy ADD COLUMN sets POLYGON;
UPDATE t_hierarchy SET sets = POLYGON(BOX(POINT(-1,lft), POINT(1, rgt)));

It needed to be a POLYGON instead of a BOX since there is a @>(POLYGON,POLYGON) function but no @>(BOX,BOX) function, and the polygon was cast from the box to create the rectangle shape required.

It outperforms the adjacency list on all descendants; outperforms it on all ancestors (not by much); performs reasonably well on all descendants up to a certain level on items with few descendants (e. g. 31415) and badly on items with many descendants (e. g. 42).

It still completes in less than 20 seconds though, which is an improvement over 1 minute.

PostgreSQL does support R-Tree indexes indeed (through GiST interface), and they can be used to improve the efficiency of the nested sets model.

Let’s create a sample table and try some of the queries that Jay proposed:

Table creation details

To make the management of the table easier, I didn’t create an additional column with the geometric representation of the nested sets, but instead just defined an index on a derived expression, so that updating lft and rgt columns would be enough to update the set.

Now, let’s see how these queries perform.

All descendants

SELECT  SUM(LENGTH(hc.stuffing)), COUNT(*)
FROM    t_hierarchy hp
JOIN    t_hierarchy hc
ON      POLYGON(BOX(POINT(-1, hc.lft), POINT(1, hc.rgt))) <@ POLYGON(BOX(POINT(-1, hp.lft), POINT(1, hp.rgt)))
WHERE   hp.id = 42

View query details

Quite fast, 213 ms.

All ancestors

SELECT  hc.id, hc.lft, hc.rgt, hc.parent
FROM    t_hierarchy hp
JOIN    t_hierarchy hc
ON      POLYGON(BOX(POINT(-1, hc.lft), POINT(1, hc.rgt))) @> POLYGON(BOX(POINT(-1, hp.lft), POINT(1, hp.rgt)))
WHERE   hp.id = 42

View query details

Extremely fast: only 10 ms.

All descendants up to a certain level

SELECT  hc.id, hc.lft, hc.rgt, hc.parent
FROM    t_hierarchy hp
JOIN    t_hierarchy hc
ON      POLYGON(BOX(POINT(-1, hc.lft), POINT(1, hc.rgt))) <@ POLYGON(BOX(POINT(-1, hp.lft), POINT(1, hp.rgt)))
WHERE   hp.id = 42
        AND
        (
        SELECT  COUNT(*)
        FROM    t_hierarchy hcp
        WHERE   POLYGON(BOX(POINT(-1, hc.lft), POINT(1, hc.rgt))) <@ POLYGON(BOX(POINT(-1, hcp.lft), POINT(1, hcp.rgt)))
        ) -
        (
        SELECT  COUNT(*)
        FROM    t_hierarchy hpp
        WHERE   POLYGON(BOX(POINT(-1, hp.lft), POINT(1, hp.rgt))) <@ POLYGON(BOX(POINT(-1, hpp.lft), POINT(1, hpp.rgt)))
        ) <= 2

View query details

This, exactly as was mentioned by Jay, is much faster than using a B-Tree index but still too slow: 20 seconds.

Analysis

The nested sets model, improved by using the R-Tree indexes, provides a way to tell if two records are in the same ancestry chain.

However, even with the R-Tree, the model provides no simple means to tell how deep is a record nested.

To check it, an R-Tree index scan should be made which would return all of the record’s ancestors, the the number of the ancestors is to be compared with that of the parent node.

For a record with lots of ancestors (which was the case for the record 42 we used in the test queries), this means that thousands of records should be checked in a nested loop, out of which only a dozen will be returned.

Ironically, for the real-world models, this type of query is most often used, and used against the records with lots of descendants it is.

Usually, when hierarchical data are stored in a database, they are presented to a user in the form of a tree view. When the user opens the catalog, the first-level entries are show; when the user clicks on expand button of an entry, all immediate children of the entry should be shown.

Since users usually start browsing from the beginning, clicking the expand buttons on the first-level or second-level entries is what happens most often. And, unfortunately, it takes the most time to execute these queries.

Adjacency list model provides a constant time solution to this problem, since fetching all immediate children requires a single index scan. This is extremely fast on showing the immediate children.

A user can also click on expand all which should just return all children of the given entry.

However, clicking on expand all on a high-level entry will return too many records, so a time to download them or represent them in the GUI will be much more than that required to fetch them out of the table. A properly written GUI usually limits the level of the records returned so that GUI remains responsive, which, it its turn, implies the same problem of filtering on level.

The low-level entries (for which it makes sense to implement expand all without any limitations) can be queried for their descendants with the R-Tree query in the nested sets model or with a recursive query in the adjacency list model almost equally fast, since low-level entries contain few records.

The same applies to selecting all ancestors. Despite the fact that the nested sets model outperforms slightly the adjacency list model on this type of query, the absolute numbers are very small and the times that both queries take are almost imperceptible to the bare eye. A hierarchy is seldom more than a dozen levels deep, and fetching each ancestor even with a recursive query requires but one unique index scan per ancestor.

However, one may still be forced to use the nested tree model. This may be the way an ORM stores its data in the database; a heavily used legacy schema too old and scary to touch; or just some obscure model which mostly requires fetching all descendants fast with an occasional need to filter on the level.

Here are some methods to deal with it.

Analytic functions

Though there is no efficient way to filter all descendants on the level, there still is a way to fetch all immediate children of a record.

If we select all records within lft and rgt of a given entry and order them by lft, the first record will be the first immediate child of the entry.

All descendants of the first child will be returned before the second child and have rgt less than that of the first child.

This means that if we record the MAX(rgt) fetched so far, it will be that of the last immediate child of the entry fetched so far:

id lft rgt MAX(rgt)
2 2 11 11
3 3 4 11
4 5 8 11
5 6 7 11
6 9 10 11
7 12 15 15
8 13 14 15

This means that each value of MAX(rgt) will correspond to exactly one immediate child; and the first entry in the recordset holding the value of MAX(rgt) will be that first child.

Several method exist to select records holding group-wise maximum in PostgreSQL. In this case, is will be best to use PostgreSQL‘s DISTINCT ON.

Here’s the query:

SELECT  DISTINCT ON (MAX(hc.rgt) OVER (ORDER BY hc.lft)) hc.id, hc.lft, hc.rgt
FROM    t_hierarchy hp
JOIN    t_hierarchy hc
ON      hc.lft > hp.lft
        AND hc.lft < hp.rgt
WHERE   hp.id = 42
ORDER BY
        MAX(hc.rgt) OVER (ORDER BY hc.lft), hc.lft

View query details

, which is reasonably fast (only 160 ms).

Using PostgreSQL 8.4 recursive abilities, this approach can be extended to select the descendants up to any level (provided as a parameter to the query).

Here’s the query to select all children and grandchildren:

WITH    RECURSIVE
        q AS
        (
        SELECT  id, lft, rgt, 1 AS lvl
        FROM    t_hierarchy
        WHERE   id = 42
        UNION ALL
        SELECT  *
        FROM    (
                SELECT  DISTINCT ON (MAX(hc.rgt) OVER (PARTITION BY q.id ORDER BY hc.lft)) hc.id, hc.lft, hc.rgt, lvl + 1
                FROM    q
                JOIN    t_hierarchy hc
                ON      hc.lft > q.lft
                        AND hc.lft < q.rgt
                WHERE   lvl <= 2
                ORDER BY
                        MAX(hc.rgt) OVER (PARTITION BY q.id ORDER BY hc.lft), hc.lft
                ) q2
        )
SELECT  *
FROM    q

View query details

This is also reasonably fast, only 432 ms. It is slower than the same adjacency list query (which completes in several milliseconds), but still is much faster than R-Tree and of course the least efficient B-Tree solutions involving COUNT(*) and can ease your life if you have to deal with a nested sets model.

To be continued.

]]> http://explainextended.com/2010/03/01/postgresql-nested-sets-and-r-tree/feed/ 0 Six degrees of separation http://explainextended.com/2010/02/27/six-degrees-of-separation/ http://explainextended.com/2010/02/27/six-degrees-of-separation/#comments Sat, 27 Feb 2010 20:00:45 +0000 Quassnoi http://explainextended.com/?p=4477 Answering questions asked on the site.

Kathy asks:

I am developing a social network site in PostgreSQL and want to find out if two people are no more than 6 friends apart.

If your site grows popular, most probably, they are not. But we better check.

On most social networks, friendship is a symmetric relationship (however, LiveJournal is a notable exception). This means that if Alice is a friend to Bob, then Bob is a friend to Alice as well.

The friendship relationship is best stored in a many-to-many link table with a PRIMARY KEY on both link fields and an additional check condition: the friend with the least id should be stored in the first column. This is to avoid storing a relationship twice: a PRIMARY KEY won’t be violated if the same record with the columns swapped will be inserted, but the check constraint will. The check constraint will also forbid storing a friend relationship to itself.

Let’s create a sample table:

Table creation details

This table stores records for 1,000,000 people having 20 friends each in average. The first column is named orestes and the second one pylades.

With new PostgreSQL 8.4 it is easy to write a recursive query that would traverse the relationship graph up to the given level and stop on the first match.

However, each recursion step requires a join, and as the number of records in the input recordset for the recusion grows with level, the joins become less and less efficient. The number of records grows exponentially, and on level 6 there will be about 20 ^ 6 = 64,000,000 records on input. This is just too much for a join with a 20,000,000 records table.

As the chain length increases, the tree diverges, the number of the records grows and it becomes more and more costly to join them with the table.

To work around this, we should use Bogdan the tunnel builder’s algorithm.

British and French governments submit a tender to build a tunnel under the Channel. Many companies apply, all demanding years of time and billions pounds of money, so their offers are refused.

One day, Bogdan drops in and offers his services.

How much money would you demand for your work?, the official asks. Me and my brother Roman are good eaters, so we will need to buy a decent meal every day. 50 pounds a day will be OK.

That’s pretty cheap; and how much time will you need? Me and my brother Roman are fast diggers; one mile a day I think we will dig.

Oh, that’s pretty fast! But how will you be able to work on such a low budget in such a short time?, the official asks out of curiosity.

That’s simple, Bogdan answers, I start digging from the British coast, my brother Roman starts digging from the French coast; the moment we meet, the work is over.

OK, says the official, but what if you don’t meet?.

No worries then: you get two tunnels for the price of one

We should do a similar thing here. Instead of traversing the 6 levels from the beginning, we will traverse just 3 levels from each side, then join the resulting recordsets and hope some matching records will be found.

Traversing only 3 levels will be quite fast; and the resulting recordsets will be of moderate size so joining them will be easy.

As an extra, we will return the shortest path from one person to the other. To do this, we will need to record the friendship chain in an array. PostgreSQL does not offer an easy way to reverse an array, so in the first recordset, we will append the friends to the array, while in the second one we will prepend them. This way, we should just concatenate the resulting tunnelsarrays.

Here’s the query:

WITH    RECURSIVE
        q1 (person, chain, lvl) AS
        (
        SELECT  123456, ARRAY[123456], 1
        UNION ALL
        SELECT  friend, chain || friend, lvl + 1
        FROM    (
                SELECT  q1.*,
                        friend
                FROM    q1
                JOIN    (
                        SELECT  orestes AS me, pylades AS friend
                        FROM    friends
                        UNION ALL
                        SELECT  pylades AS me, orestes AS friend
                        FROM    friends
                        ) f
                ON      person = me
                WHERE   lvl <= 3
                ) qo
        ),
        q2 (person, chain, lvl) AS
        (
        SELECT  654321, ARRAY[654321], 1
        UNION ALL
        SELECT  friend, friend || chain, lvl + 1
        FROM    (
                SELECT  q2.*,
                        friend
                FROM    q2
                JOIN    (
                        SELECT  orestes AS me, pylades AS friend
                        FROM    friends
                        UNION ALL
                        SELECT  pylades AS me, orestes AS friend
                        FROM    friends
                        ) f
                ON      person = me
                WHERE   lvl <= 3
                ) qo
        )
SELECT  (q1.chain || q2.chain[2:q2.lvl])::TEXT AS chain
FROM    q1
JOIN    q2
ON      q2.person = q1.person
ORDER BY
        q1.lvl + q2.lvl
LIMIT 1

View query details

Note that the anchor part can not be used more than once in a recursive expression. To work around that, we had to join it to a derived table (a UNION ALL of two copies of the table with the columns swapped). However, PostgreSQL‘s optimizer was smart enough to push the join predicate into the derived table and distribute the queries so that each part uses a corresponding index efficiently. This helps to traverse the tree and build the recordsets from both ends.

Each of the recordsets has only about 8,000 records, so scanning and joining them is very fast.

The whole query takes just a little longer than 0.5 seconds.

Hope that helps.

I’m always glad to answer the questions regarding database queries.

Ask me a question

]]> http://explainextended.com/2010/02/27/six-degrees-of-separation/feed/ 0 Sargability of monotonic functions: example http://explainextended.com/2010/02/23/sargability-of-monotonic-functions-example/ http://explainextended.com/2010/02/23/sargability-of-monotonic-functions-example/#comments Tue, 23 Feb 2010 20:00:25 +0000 Quassnoi http://explainextended.com/?p=4407 In my previous article I presented a proposal to add sargability of monotonic functions into the SQL engines.

In a nutshell: a monotonic function is a function that preserves the order of the argument so that it gives the larger results for the larger values of the argument. It is easy to prove that a B-tree with each key replaced by the result of the function will remain the valid B-Tree and hence can be used to search for ranges of function results just like it is used to search for ranges of values.

With a little effort, a B-Tree can also be used to search for the ranges of piecewise monotonic functions: those whose domain can be split into a number of continuous pieces with the function being monotonic within each piece (but it may be not monotonic and even not continuous across the pieces).

In this article, I’ll demonstrate the algorithm to do that (implemented in pure SQL on PostgreSQL).

I will show how the performance of simple query 

SELECT  *
FROM    t_sine
WHERE   SIN(value) BETWEEN 0.1234 AND 0.1235

could be improved if the sargability of monotonic functions had been implemented in the optimizer.

To do this, I will create a sample table:

Table creation details

This table contains 1,000,000 records with value randomly distributed from 0 to 101.

To select the records we need, we can use a very simple and straightforward query:

SELECT  *
FROM    t_sine
WHERE   SIN(value) BETWEEN 0.4452 AND 0.4453

View query details

which returns 31 records in 3.25 seconds.

The query uses a full table scan with the filter applied to each record.

Let’s try to improve it.

Function description

According to the notation I proposed in the previous article, the monotony of the function SIN() should be described as this:

SIN(arg FLOAT) MONOTONIC PIECEWISE
DEFINED BY FLOOR(arg / PI() + 0.5)
CASE PIECE % 2
WHEN 0 THEN DECREASING INVERSE PIECE * PI() + ASIN(RESULT)
ELSE INCREASING INVERSE PIECE * PI() - ASIN(RESULT)
END

This means that:

  1. The function is piecewise monotonic,

  2. The pieces are defined by the function FLOOR(arg / PI() + 0.5) (which essentially returns the number of the half-wave the argument belongs too),

  3. The function monotony varies depending on the piece,

  4. On odd pieces, the function increases,

  5. On even pieces, the function decreases.

  6. A single inverse expression is provided for each monotony

Note that mathematically the function is strictly monotonic on each of its pieces. However, due to the rounding errors, different arguments can yield same function results, so the function value may map back to a range of the arguments rather than a single value.

In theory, it is possible to write a single expression which would map the function’s result to the pair of values defining the beginning and the end of such a range. However, the expression would be quite complex. So for illustration purposes I’ll make do with a single inverse function that yields an approximation of the back mapping. To find the exact range, some extra effort will be required.

Building the pieces

The function is piecewise monotonic and the pieces are defined by a function. For the pieces to be continuous, the function that defines them should be itself monotonic over all its domain.

The function that defines the pieces is FLOOR(arg / PI() + 0.5).

It is a superposition of the three functions:

  • OPERATOR_DIVISION(arg1 FLOAT, arg2 FLOAT)
    MONOTONIC OVER (arg1)
    CASE WHEN arg2 > 0 THEN INCREASING INVERSE RESULT * arg2
    WHEN arg2 = 0 THEN UNDEFINED
    WHEN arg2 < 0 THEN DECREASING INVERSE RESULT * arg2
    END

  • OPERATOR_PLUS(arg1 FLOAT, arg2 FLOAT) MONOTONIC
    OVER (arg1) STRICTLY INCREASING INVERSE RESULT - arg2,
    OVER (arg2) STRICTLY INCREASING INVERSE RESULT - arg1

  • FLOOR(arg FLOAT) MONOTONIC INCREASING
    INVERSE
    FROM RESULT EXACT
    TO RESULT + 1 EXACT EXCLUDE

which are, given the values of the constants provided in the secondary arguments, are increasing over the argument. As we know from math, a superposition of monotonic functions is also monotonic.

Each function is defined with a single inverse condition which maps the result of the function back to a value near the range of the arguments yielding the result. The exact range has to be sought for using index seek over (hopefully) not too many records.

Sequentially applying the inverse expressions of each of the constituent functions to the result of the piece defining function, we get the following inverse expression for the latter:

  1. FLOOR(OPERATOR_PLUS(OPERATOR_DIVISION(arg, PI()), 0.5)) = PIECE
  2. OPERATOR_PLUS(OPERATOR_DIVISION(arg, PI()), 0.5) ∈ [ PIECE, PIECE + 1 )
  3. OPERATOR_DIVISION(arg, PI()) ∈ [ ≈(PIECE - 0.5), ≈((PIECE + 1) - 0.5) ]
  4. arg ∈ [ ≈((PIECE - 0.5) * PI()), ≈(((PIECE + 1) - 0.5) * PI()) ]

For each piece, we how have a pair of values approximately defining the range of values belonging to the piece.

To find out the exact bounds, we need to do the following:

  1. Calculate the piece for the minimal value

  2. Find the approximate upper bound for the piece.

  3. Scanning the keys to the left, find the rightmost key to the left of the upper bound that belongs to the current (or previous) piece.

  4. Scanning the keys to the right, find the first key of the next piece.

  5. Scanning a single key to the left, find the last key of the current piece.

  6. Recursively repeat steps 1 to 5, taking the first value the next piece calculated on step 4 as a seed for the step 1, until step 4 fails (which means that the pieces are over).

This procedure guarantees that we always get the correct bounds even with the inexact inverse value, since it correctly handles both overflow and underflow of the inverse value, as show on the pictures below:

Overflow

Underflow

Here's a query that selects the first and the last key of each piece:

View the query

Despite being huge in size, the query is very efficient and completes in only 48 ms.

Locating values within the pieces

Now, when we have the exact bounds of each piece, we need to locate the records within each piece.

Since we don't have exact inverse function here, the basic idea is the same as above: given the approximate inverse, locate the exact bound using the iterative approach:

  1. Locate the first key to the left of the inverse which yields the function result less than the one sought for, up to the first key of the piece. Should this search fail, the first key of the piece is the lower bound.
  2. Locate the first key to the right of that found on the previous step that yields the function value equal to or greater than the one sought for, up to the last key of the piece. Return NULL should it fail

Since we have an inclusive range here, we don't need the third step (final scan to the left to find the rightmost least value) that we used when searching for the pieces.

This algorithm searches for the lower bound; to search for the upper bound, we just need to inverse both directions and tests (left becomes right, less becomes greater etc).

Since the monotony of the function varies from piece to piece, we should take this into account. For the pieces where the function's monotony is DECREASING we should swap the order of the bounds: the upper bound of the expression becomes the lower bound or the range of values and vice versa. This can be handled merely by substituting the conditions into the very same CASE expression that defines the monotony.

When we locate the upper and the lower bounds for each piece, we should just join t_sine on the following condition:

ON value BETWEEN llimit and ulimit

It can happen so that the lower bound found by the algorithm exceeds the upper bound. This is a perfectly normal situation meaning that no keys match the condition and the range diverged. BETWEEN predicate will handle this.

It also can happen that one of the bounds is a NULL. This is also a valid situation, meaning that no value within the piece exceeds the lower bound (or falls short of the upper one). BETWEEN will also take care of it.

Final query

And here's the final query. Be ready, you'll have to spin your mouse wheel a lot:

WITH    RECURSIVE
        d AS (
        SELECT  piece,
                minv,
                COALESCE(
                (
                SELECT  value
                FROM    t_sine
                WHERE   value < nv[1]
                ORDER BY
                        value DESC
                LIMIT 1
                ),
                (
                SELECT  MAX(value)
                FROM    t_sine
                )
                ) AS maxv,
                nv[1] AS nextv,
                nv[2] AS nextpiece
        FROM    (
                SELECT  minv, piece,
                        (
                        SELECT  ARRAY[value, FLOOR(value / PI() + 0.5)]
                        FROM    t_sine
                        WHERE   value >
                                (
                                SELECT  value
                                FROM    t_sine
                                WHERE   value <= ((piece + 1) - 0.5) * PI()
                                        AND FLOOR(value / PI() + 0.5) <= piece
                                ORDER BY
                                        value DESC
                                LIMIT 1
                                )
                        ORDER BY
                                value
                        LIMIT 1
                        ) nv
                FROM    (
                        SELECT  minv, FLOOR(minv / PI() + 0.5) AS piece
                        FROM    (
                                SELECT  MIN(value) AS minv
                                FROM    t_sine
                                ) q
                        ) q2
                ) q3
        UNION ALL
        SELECT  piece,
                minv,
                COALESCE(
                (
                SELECT  value
                FROM    t_sine
                WHERE   value < nv[1]
                ORDER BY
                        value DESC
                LIMIT 1
                ),
                (
                SELECT  MAX(value)
                FROM    t_sine
                )
                ) AS maxv,
                nv[1] AS nextv,
                nv[2] AS nextpiece
        FROM    (
                SELECT  minv, piece,
                        (
                        SELECT  ARRAY[value, FLOOR(value / PI() + 0.5)]
                        FROM    t_sine
                        WHERE   value >
                                (
                                SELECT  value
                                FROM    t_sine
                                WHERE   value <= ((piece + 1) - 0.5) * PI()
                                        AND FLOOR(value / PI() + 0.5) <= piece
                                ORDER BY
                                        value DESC
                                LIMIT 1
                                )
                        ORDER BY
                                value
                        LIMIT 1
                        ) nv
                FROM    (
                        SELECT  nextv AS minv, nextpiece AS piece
                        FROM    d
                        WHERE   nextpiece IS NOT NULL
                        ) q2
                ) q3
        )
SELECT  l.*, s.*, SIN(value)
FROM    (
        SELECT  minv, maxv,
                CASE piece::INTEGER % 2
                WHEN 0 THEN
                        (
                        SELECT  value
                        FROM    t_sine
                        WHERE   value >=
                                COALESCE(
                                (
                                SELECT  value
                                FROM    t_sine
                                WHERE   value <= LEAST(piece * PI() + ASIN(0.4452), maxv)
                                        AND value >= minv
                                        AND SIN(value) < 0.4452
                                ORDER BY
                                        value DESC
                                LIMIT 1
                                ),
                                minv
                                )
                                AND value <= maxv
                                AND SIN(value) >= 0.4452
                        ORDER BY
                                value
                        LIMIT 1
                        )
                ELSE
                        (
                        SELECT  value
                        FROM    t_sine
                        WHERE   value >=
                                COALESCE(
                                (
                                SELECT  value
                                FROM    t_sine
                                WHERE   value <= LEAST(piece * PI() - ASIN(0.4453), maxv)
                                        AND value >= minv
                                        AND SIN(value) > 0.4453
                                ORDER BY
                                        value DESC
                                LIMIT 1
                                ),
                                minv
                                )
                                AND value <= maxv
                                AND SIN(value) <= 0.4453
                        ORDER BY
                                value
                        LIMIT 1
                        )
                END AS llimit,
                CASE piece::INTEGER % 2
                WHEN 0 THEN
                        (
                        SELECT  value
                        FROM    t_sine
                        WHERE   value <=
                                COALESCE(
                                (
                                SELECT  value
                                FROM    t_sine
                                WHERE   value >= GREATEST(piece * PI() + ASIN(0.4453), minv)
                                        AND value <= maxv
                                        AND SIN(value) > 0.4453
                                ORDER BY
                                        value
                                LIMIT 1
                                ),
                                maxv
                                )
                                AND value >= minv
                                AND SIN(value) <= 0.4453
                        ORDER BY
                                value DESC
                        LIMIT 1
                        )
                ELSE
                        (
                        SELECT  value
                        FROM    t_sine
                        WHERE   value <=
                                COALESCE(
                                (
                                SELECT  value
                                FROM    t_sine
                                WHERE   value >= GREATEST(piece * PI() - ASIN(0.4452), minv)
                                        AND value <= maxv
                                        AND SIN(value) < 0.4452
                                ORDER BY
                                        value
                                LIMIT 1
                                ),
                                maxv
                                )
                                AND value >= minv
                                AND SIN(value) >= 0.4452
                        ORDER BY
                                value DESC
                        LIMIT 1
                        )
                END AS ulimit
        FROM    d
        ) l
JOIN    t_sine s
ON      value BETWEEN llimit AND ulimit

View query details

This 200-line monster completes in only 110 ms, or 30 times as fast as the original 3-liner:

SELECT  *
FROM    t_sine
WHERE   SIN(value) BETWEEN 0.1234 AND 0.1235

, yielding the same results.

Summary

This example was to demonstrate feasibility of the B-Tree indexes to be used in a search for the predicates involving monotonic functions and the performance gain achieved.

The performance gain is over 30 times for a table that fits completely into the cache, and will increase with the number of the cache misses increases.

The SQL implementation of the algorithm is in fact not optimal, since iterative searches for the value boundaries are implemented as the subqueries. Each subquery requires reentering the B-Tree and traversing it starting from the root. The native algorithm working within the optimizer could avoid this by caching the key position in the index and issuing next_key / prev_key commands, which would improve the algorithm yet more.

Sargability of the monotonic functions, as shown above, can help to make the queries like the one described in this article much more legible, maintainable and efficient.

]]> http://explainextended.com/2010/02/23/sargability-of-monotonic-functions-example/feed/ 0 Searching for arbitrary portions of a date http://explainextended.com/2010/02/02/searching-for-arbitrary-portions-of-a-date/ http://explainextended.com/2010/02/02/searching-for-arbitrary-portions-of-a-date/#comments Tue, 02 Feb 2010 20:00:13 +0000 Quassnoi http://explainextended.com/?p=4132 From Stack Overflow:

I have a Ruby on Rails application with a PostgreSQL database; several tables have created_at and updated_at timestamp attributes.

When displayed, those dates are formatted in the user’s locale; for example, the timestamp 2009-10-15 16:30:00.435 becomes the string 15.10.2009 – 16:30 (the date format for this example being dd.mm.yyyy - hh.mm).

The requirement is that the user must be able to search for records by date, as if they were strings formatted in the current locale.

For example, searching for 15.10.2009 would return records with dates on October 15th 2009; searching for 15.10 would return records with dates on October 15th of any year, searching for 15 would return all dates that match 15 (be it day, month or year).

The simplest solution would be just retrieve the locale string from the client, format the dates according to that string and search them using LIKE or ~ operators (the latter, as we all know, searches for POSIX regular expressions).

However, this would be not very efficient.

Let’s create a sample table and see:

Table creation details

This query contains 1,000,000 records with random timestamps spanning more than 114 years.

Assuming that the client’s date format is set to dd.mm.yy hh24.mi.ss, let’s try to select the number of records that satisfy this string: '20.12'. We are also assuming that the beginning of string and the end of string are the field separators as well:

SELECT  COUNT(*)
FROM    t_dates
WHERE   TO_CHAR(date, 'dd.mm.yy hh24.mi.ss') ~ E'(^|[^\\d])20\\.12([^\\d]|$)'
count
4235
1 row fetched in 0.0001s (7.9138s) 
Aggregate  (cost=47262.58..47262.59 rows=1 width=0)
  ->  Seq Scan on t_dates  (cost=0.00..47258.97 rows=1444 width=0)
        Filter: (to_char(date, 'dd.mm.yy hh24.mi.ss'::text) ~ '(^|[^\\d])20\\.12([^\\d]|$)'::text)

This query runs for almost 8 seconds. Let’s see which values does it return:

SELECT  id, date
FROM    t_dates
WHERE   TO_CHAR(date, 'dd.mm.yy hh24.mi.ss') ~ E'(^|[^\\d])20\\.12([^\\d]|$)'
ORDER BY
        MD5(id::TEXT)
LIMIT 10
id date
622924 1939-01-10 20:12:31
781217 1920-12-20 07:11:26
501772 1952-11-05 20:12:03
956539 1900-12-20 04:57:58
523679 1950-05-08 01:20:12
141308 1993-12-20 04:07:37
648220 1936-02-21 20:12:30
236980 1983-01-20 20:12:29
413051 1962-12-20 13:20:54
323566 1973-03-06 02:20:12

We see the matches on day-month, hour-minute and minute-second. There are no month-year or year-hour matches, since there is no 20th month and the year-hour separator is not a period.

The query seems to return correct values but is quite slow.

To improve this query we can use PostgreSQL‘s GIN indexing abilities.

A GIN index is a way to index one record with several keys.

A plain index is a B-Tree structure that stores the pointers to the records (the ctid‘s) in the leaf nodes. Such an index only accepts a single expression as a key and builds a single sort order over these expressions, so each record can be pointed to at most once. There is a one-to-many mapping between keys and records: a key can point to many records, but a record can be pointed to by at most one key.

A GIN index, on the other hand, accepts an array of expressions as a parameter and uses each element of the array as a key. This way, the mapping becomes many-to-many: each key can point to many records, and each record can be pointed to by many keys.

Usually, GIN indexes are used for FULLTEXT indexing: the piece of text stored in a record is split into the separate words and each word is indexed separately so that search for any word can be performed using the index.

However, in PostgreSQL, GIN indexes support integer arrays as well. And we can use this support to improve our query.

As many of you may have noted, I created a GIN index in the table creation script. Here’s how it looks:

CREATE INDEX ix_dates_parts ON t_dates
USING GIN((
        ARRAY[
        DATE_PART('year', date)::INTEGER,
        DATE_PART('year', date)::INTEGER % 100,
        DATE_PART('month', date)::INTEGER,
        DATE_PART('day', date)::INTEGER,
        DATE_PART('hour', date)::INTEGER,
        (DATE_PART('hour', date)::INTEGER + 1) % 12 + 1,
        DATE_PART('minute', date)::INTEGER,
        DATE_PART('second', date)::INTEGER
        ]
        ));

Each record is split into an array of 8 integers, each representing a certain portion of a date which are normally used in the date formatting options. 6 of them just represent date parts, and there are two extra integers that represent a 2-digit year and an AM/PM hour.

This way, any record in year 2010 gets indexed with both 2010 and 10, and any record with hour 19 gets indexed with both 19 and 7.

This covers most formatting options, and can be changed to include less used ones.

To make use of this index we should provide an additional predicate in the WHERE clause. This predicate will take a user-provided array as an input and search for the records that contain all elements of the user-provided array in the date parts.

Here’s how our query looks now:

SELECT  COUNT(*)
FROM    t_dates
WHERE   TO_CHAR(date, 'dd.mm.yy hh24.mi.ss') ~ E'(^|[^\\d])20\\.12([^\\d]|$)'
        AND ARRAY[20, 12] <@ ARRAY[
        DATE_PART('year', date)::INTEGER,
        DATE_PART('year', date)::INTEGER % 100,
        DATE_PART('month', date)::INTEGER,
        DATE_PART('day', date)::INTEGER,
        DATE_PART('hour', date)::INTEGER,
        (DATE_PART('hour', date)::INTEGER + 1) % 12 + 1,
        DATE_PART('minute', date)::INTEGER,
        DATE_PART('second', date)::INTEGER
        ]
count
4235
1 row fetched in 0.0001s (0.2366s) 
Aggregate  (cost=3595.90..3595.91 rows=1 width=0)
  ->  Bitmap Heap Scan on t_dates  (cost=104.76..3595.90 rows=1 width=0)
        Recheck Cond: ('{20,12}'::integer[] <@ ARRAY[(date_part('year'::text, date))::integer, ((date_part('year'::text, date))::integer % 100), (date_part('month'::text, date))::integer, (date_part('day'::text, date))::integer, (date_part('hour'::text, date))::integer, ((((date_part('hour'::text, date))::integer + 1) % 12) + 1), (date_part('minute'::text, date))::integer, (date_part('second'::text, date))::integer])
        Filter: (to_char(date, 'dd.mm.yy hh24.mi.ss'::text) ~ '(^|[^\\d])20\\.12([^\\d]|$)'::text)
        ->  Bitmap Index Scan on ix_dates_parts  (cost=0.00..104.76 rows=1000 width=0)
              Index Cond: ('{20,12}'::integer[] <@ ARRAY[(date_part('year'::text, date))::integer, ((date_part('year'::text, date))::integer % 100), (date_part('month'::text, date))::integer, (date_part('day'::text, date))::integer, (date_part('hour'::text, date))::integer, ((((date_part('hour'::text, date))::integer + 1) % 12) + 1), (date_part('minute'::text, date))::integer, (date_part('second'::text, date))::integer])

This returns the same records but does it 40 times as fast, since the GIN index is used for coarse filtering and the fine-filtering operator is applied to selected results only.

For the index to work, the expression we used to create the index should be provided verbatim to the right side of the <@ (contains) operator.

This solution, however, does not cover all possible conditions: a client can use less obvious formats. However, with a proper design this should not be a problem. The client application should just ignore the parts which are formatted in a way not suitable for the index and do not put them into the array (but of course leave them in the regular expression).

This makes the index less selective but still usable, and the query performance will still be improved greatly.

]]> http://explainextended.com/2010/02/02/searching-for-arbitrary-portions-of-a-date/feed/ 0 PostgreSQL: Selecting records holding group-wise maximum http://explainextended.com/2009/11/26/postgresql-selecting-records-holding-group-wise-maximum/ http://explainextended.com/2009/11/26/postgresql-selecting-records-holding-group-wise-maximum/#comments Thu, 26 Nov 2009 20:00:25 +0000 Quassnoi http://explainextended.com/?p=3777 Continuing the series on selecting records holding group-wise maximums:

How do I select the whole records, grouped on grouper and holding a group-wise maximum (or minimum) on other column?

In this article, I’ll describe several ways to do this in PostgreSQL 8.4.

PostgreSQL 8.4 syntax is much richer than that of MySQL. The former can use the analytic functions, recursive CTE‘s and proprietary syntax extensions, all of which can be used for this task.

Let’s create a sample table:

Table creation details

This table has 1,000,000 records:

  • id is the PRIMARY KEY
  • orderer is filled with random values from 1 to 10
  • glow is a low cardinality grouping field (10 distinct values)
  • ghigh is a high cardinality grouping field (10,000 distinct values)
  • stuffing is an asterisk-filled VARCHAR(200) column added to emulate payload of the actual tables

Analytic functions

PostgreSQL 8.4 supports analytic functions. These functions extend the aggregate abilities: they work on the groups rather than on the individual records, but return their values to each individual record instead of shrinking the set. For instance, ROW_NUMBER enumerates the records within the group according to the ordering condition, and DENSE_RANK enumerates distinct values of the ordering column (it assigns same number to the records with the same value of the ordering column).

Let’s make a query to select the records holding the group-wise maximums of id. Since id is a PRIMARY KEY we don’t have to worry about the ties.

Here’s the query:

SELECT  id, orderer, glow, ghigh
FROM    (
        SELECT  *, ROW_NUMBER() OVER (PARTITION BY glow ORDER BY id) AS rn
        FROM    t_distinct
        ) q
WHERE   rn = 1

View query details

This works, but is very inefficient (more than 12 seconds). PostgreSQL chooses the sorting in this case but it is not very good in sorting the tables with large rows.

This can be improved by making PostgreSQL to use the index which covers both columns and then join the id (which is also covered by the index):

SELECT  di.id, di.orderer, di.glow, di.ghigh
FROM    (
        SELECT  id, ROW_NUMBER() OVER (PARTITION BY glow ORDER BY id) AS rn
        FROM    t_distinct d
        ) dd
JOIN    t_distinct di
ON      di.id = dd.id
WHERE   rn = 1

View query details

This is much faster (4 s) but there is still much space for improvement.

If we wish to order by orderer we need to define a method to resolve ties.

Using the same approach we can return all records with ties:

SELECT  COUNT(*), SUM(id)
FROM    (
        SELECT  *, DENSE_RANK() OVER (PARTITION BY glow ORDER BY orderer) AS dr
        FROM    t_distinct d
        ) dd
WHERE   dr = 1

View query details

, or resolve ties by return the record with the maximum id among those holding the minimum value of the orderer:

SELECT  di.id, di.orderer, di.glow, di.ghigh
FROM    (
        SELECT  id, ROW_NUMBER() OVER (PARTITION BY glow ORDER BY orderer, id DESC) AS rn
        FROM    t_distinct d
        ) dd
JOIN    t_distinct di
ON      di.id = dd.id
WHERE   rn = 1

View query details

As you can see, all these queries are elegant but rather inefficient.

Using DISTINCT ON

PostgreSQL implements another way to return the whole records holding group-wise maximums or minimums.

By using a special clause, DISTINCT ON, we can return records holding only the distinct values of the certain columns. For this to work correctly, one needs to define an ORDER BY condition in addition to DISTINCT ON, with the leading expressions being the same as those using in DISTINCT ON. This guarantees that all the records belonging to each group would be consecutive if not for the DISTINCT ON clause.

DISTINCT ON is applied after the ORDER BY condition. It just returns the first record from each group, skipping the others. This is very easy to do: return a record if the grouping expression changed from the previous row; don’t return if it didn’t.

This is quite similar to MySQL‘s extension for GROUP BY, but, unlike MySQL, this solution guarantees correct order and the fact that all values returned will be taken from a single record.

This query cannot be used to return all records with ties (since the values of the grouping column won’t be distinct), but it will work if the ties are impossible (as in selecting a maximum id), or if a correct condition for resolving ties is provided.

Here’s the query to return records holding MAX(id):

SELECT  DISTINCT ON (glow) id, orderer, glow, ghigh
FROM    t_distinct
ORDER BY
        glow, id DESC

View query details

And here’s the one to return the MAX(id) within the MIN(orderer):

SELECT  DISTINCT ON (glow) id, orderer, glow, ghigh
FROM    t_distinct
ORDER BY
        glow, orderer, id DESC

View query details

This is more efficient than the window function. However, both queries still take 4 seconds. This is almost 40 times as much as the same queries in MySQL , even without any improvements.

Unlike MySQL, PostgreSQL does not implement loose index scan which would allow to jump over the distinct index records. However, it can be emulated using recursive CTE‘s.

Recursive CTE’s to emulate loose index scan

The main idea here is simple:

  • In the anchor part of the CTE take the lowest value of the key
  • In the recursive part of the CTE take the next value of the key by using > or < operators along with the ORDER BY and LIMIT 1

PostgreSQL‘s syntax allows compacting a whole table record into a single field (which can be exploded later). This will allow us to avoid joins by placing the whole recursive part into a subquery which will use the index efficiently.

Here’s the query to return records holding group-wise MAX(id):

WITH    RECURSIVE rows AS
        (
        SELECT  d
        FROM    (
                SELECT  d
                FROM    t_distinct d
                ORDER BY
                        glow DESC, id DESC
                LIMIT 1
                ) q
        UNION ALL
        SELECT  (
                SELECT  di
                FROM    t_distinct di
                WHERE   di.glow < (r.d).glow
                ORDER BY
                        di.glow DESC, di.id DESC
                LIMIT 1
                )
        FROM    rows r
        WHERE   d IS NOT NULL
        )
SELECT  (d).id, (d).orderer, (d).glow, (d).ghigh
FROM    rows
WHERE   d IS NOT NULL
id orderer glow ghigh
1000000 6 10 10000
999999 8 9 9999
999998 2 8 9998
999997 3 7 9997
999996 8 6 9996
999995 4 5 9995
999994 5 4 9994
999993 6 3 9993
999992 3 2 9992
999991 5 1 9991
10 rows fetched in 0.0005s (0.0049s) 
CTE Scan on rows  (cost=232.94..234.96 rows=100 width=32)
  Filter: (d IS NOT NULL)
  CTE rows
    ->  Recursive Union  (cost=0.00..232.94 rows=101 width=32)
          ->  Subquery Scan q  (cost=0.00..2.24 rows=1 width=32)
                ->  Limit  (cost=0.00..2.23 rows=1 width=40)
                      ->  Index Scan Backward using ix_distinct_glow_id on t_distinct d  (cost=0.00..2231423.39 rows=1000000 width=40)
          ->  WorkTable Scan on rows r  (cost=0.00..22.87 rows=10 width=32)
                Filter: (r.d IS NOT NULL)
                SubPlan 1
                  ->  Limit  (cost=0.00..2.27 rows=1 width=40)
                        ->  Index Scan Backward using ix_distinct_glow_id on t_distinct di  (cost=0.00..755578.45 rows=333333 width=40)
                              Index Cond: (glow < ($1).glow)

As you can see, this query takes only 5 ms, next to instant. This is because on each iteration step, the whole record can be returned in a single index seek for the first value of the key which is greater than the previous value.

If we wanted to resolve the ties with more complex conditions, the query would become a little more complex too.

Let’s consider the query to resolve ties by selecting MAX(id) within the MIN(orderer), just like in the previous example.

The indexes we created order all columns in the same directions: (glow ASC, orderer ASC, id ASC). Of course, the whole index could be used as well if all directions were reversed: (glow DESC, orderer DESC, id DESC).

However, if only some of the directions are reversed, like in (orderer DESC, id ASC) (which is what we need here), the index cannot be used for ordering anymore.

The same problem was mentioned in one of the previous articles on selecting records holding group-wise maximums in MySQL. And this is the reason for the MAX(id) being less efficient than MIN(id) with a loose index scan (which is described in more details in the article aforementioned). However, MySQL deals with it automatically, while we need to implement this with our own hands.

To do this, we should need to use the same trick as we did in MySQL: select the MIN(orderer) and MAX(id) within this orderer in two different queries which would use two different index seeks, each in the appropriate direction.

Here’s the query:

WITH    RECURSIVE groups AS
        (
        SELECT  d
        FROM    (
                SELECT  d
                FROM    t_distinct d
                ORDER BY
                        glow, orderer
                LIMIT 1
                ) q
        UNION ALL
        SELECT  (
                SELECT  di
                FROM    t_distinct di
                WHERE   di.glow > (g.d).glow
                ORDER BY
                        di.glow, di.orderer
                LIMIT 1
                )
        FROM    groups g
        WHERE   d IS NOT NULL
        ),
        rows AS
        (
        SELECT  (
                SELECT  di
                FROM    t_distinct di
                WHERE   di.glow = (g.d).glow
                        AND di.orderer = (g.d).orderer
                ORDER BY
                        id DESC
                LIMIT 1
                ) di
        FROM    groups g
        WHERE   d IS NOT NULL
        )
SELECT  (di).id, (di).orderer, (di).glow, (di).ghigh
FROM    rows
id orderer glow ghigh
999881 1 1 9881
999892 1 2 9892
999923 1 3 9923
999984 1 4 9984
999955 1 5 9955
999936 1 6 9936
999827 1 7 9827
999848 1 8 9848
999829 1 9 9829
999930 1 10 9930
10 rows fetched in 0.0005s (0.0058s) 
CTE Scan on rows  (cost=588.47..590.47 rows=100 width=32)
  CTE groups
    ->  Recursive Union  (cost=0.00..243.79 rows=101 width=32)
          ->  Subquery Scan q  (cost=0.00..2.35 rows=1 width=32)
                ->  Limit  (cost=0.00..2.34 rows=1 width=40)
                      ->  Index Scan using ix_distinct_glow_orderer_id on t_distinct d  (cost=0.00..2343052.63 rows=1000000 width=40)
          ->  WorkTable Scan on groups g  (cost=0.00..23.94 rows=10 width=32)
                Filter: (g.d IS NOT NULL)
                SubPlan 1
                  ->  Limit  (cost=0.00..2.37 rows=1 width=40)
                        ->  Index Scan using ix_distinct_glow_orderer_id on t_distinct di  (cost=0.00..791389.47 rows=333333 width=40)
                              Index Cond: (glow > ($1).glow)
  CTE rows
    ->  CTE Scan on groups g  (cost=0.00..344.68 rows=100 width=32)
          Filter: (d IS NOT NULL)
          SubPlan 3
            ->  Limit  (cost=0.00..3.43 rows=1 width=36)
                  ->  Index Scan Backward using ix_distinct_glow_orderer_id on t_distinct di  (cost=0.00..38073.13 rows=11111 width=36)
                        Index Cond: ((glow = ($3).glow) AND (orderer = ($3).orderer))

We see both Index Scan and Index Scan Backward in the plan above. The first one finds the MIN(orderer), the second one finds the MAX(id) within the previously found value of the orderer.

Note that unlike MySQL, in PostgreSQL it’s enough to use just a single ORDER BY id DESC condition in the subquery which selects the top id within the records with the lowest orderer. PostgreSQL‘s optimizer is smart enough to pick the correct index (that is the index on (glow, orderer, id)) to serve this query.

This query also takes only 5 ms.

Summary

Unlike MySQL, PostgreSQL implements several clean and documented ways to select the records holding group-wise maximums, including window functions and DISTINCT ON.

However to the lack of the loose index scan support by the PostgreSQL‘s optimizer and the less efficient usage of indexes in PostgreSQL, the queries using these function take too long.

To word around these problems and improve the queries against the low cardinality grouping conditions, a certain solution described in the article should be used.

This solution uses recursive CTE‘s to emulate loose index scan and is very efficient if the grouping columns have low cardinality.

To be continued.

]]> http://explainextended.com/2009/11/26/postgresql-selecting-records-holding-group-wise-maximum/feed/ 0 Recursive CTE’s: PostgreSQL http://explainextended.com/2009/11/23/recursive-ctes-postgresql/ http://explainextended.com/2009/11/23/recursive-ctes-postgresql/#comments Mon, 23 Nov 2009 20:00:24 +0000 Quassnoi http://explainextended.com/?p=3721 In the previous article on recursive CTE‘s in SQL Server I demonstrated that they are not really set-based.

SQL Server implements the recursive CTE‘s syntax, but forbids all operations that do not distribute over UNION ALL, and each recursive step sees only a single record from the previous step.

Now, let’s check the same operations in PostgreSQL 8.4.

To do this, we well write a query that selects only the very first branch of a tree: that is, each item would be the first child of its parent. To do this, we should select the item that would be the first child of the root, the select the first child of that item etc.

This is a set-based operation.

Oracle‘s CONNECT BY syntax, despite being set-based, offers some limited set-based capabilities: you can use ORDER SIBLINGS BY clause to define the order in which the siblings are returned. However, this would require some additional work to efficiently return only the first branch.

In a true set-based system, this is much more simple.

Let’s create a sample table:

CREATE TABLE t_recursive (
        id INT NOT NULL PRIMARY KEY,
        parent INT NOT NULL,
        orderer INT NOT NULL,
        data VARCHAR(100) NOT NULL
        );

CREATE INDEX ix_recursive_parent_orderer ON t_recursive (parent, orderer);

SELECT  SETSEED(0.20091123);

INSERT
INTO    t_recursive
SELECT  s, (s - 1) / 5, FLOOR(RANDOM() * 10000), 'Item ' || s
FROM    generate_series(1, 1000000) s;

This table contains 1,000,000 records and implements an adjacency tree hierarchy.

Each item has at most 5 children and a randomly filled column, orderer, which defines its order along its siblings.

Now, let’s try to make a query that would select the first branch, the order being defined by the value provided in orderer.

To do this, we should make the anchor step to return the first child of the root item, and the recursive steps to return the first child of the previously returned item.

To return the first child of a given parent in the orderer order, we can use PostgreSQL‘s DISTINCT ON functionality.

This is the same as DISTINCT, but can return the whole row rather than a column DISTINCT is being applied to.

If two rows share the value of the column DISTINCT ON is being applied to, only one of the rows will be returned. Which row will it be is defined by the ORDER BY clause.

This solves the problems like return a single row that holds group-wise maximum.

This query:

SELECT  DISTINCT ON (parent) *
FROM    t_recursive
ORDER BY
        parent, orderer

is the same as this one:

SELECT  (q.r).*
FROM    (
        SELECT  r, ROW_NUMBER() OVER (PARTITION BY grouper ORDER BY orderer) AS rn
        FROM    t_recursive r
        ) q
WHERE   rn = 1

, but more legible and in some cases more efficient.

To apply this clause to the task we need, we should just use DISTINCT ON (parent) in both anchor and recursive parts:

WITH    RECURSIVE
        rows AS
        (
        SELECT  *
        FROM    (
                SELECT  DISTINCT ON (parent) *
                FROM    t_recursive
                WHERE   parent = 0
                ORDER BY
                        parent, orderer
                ) q
        UNION ALL
        SELECT  *
        FROM    (
                SELECT  DISTINCT ON (c.parent) c.*
                FROM    rows r
                JOIN    t_recursive c
                ON      c.parent = r.id
                ORDER BY
                        c.parent, c.orderer
                ) q2
        )
SELECT  *
FROM    rows
id parent orderer data
3 0 1686 Item 3
19 3 3370 Item 19
98 19 42 Item 98
492 98 1762 Item 492
2464 492 2295 Item 2464
12322 2464 2050 Item 12322
61614 12322 768 Item 61614
308074 61614 1925 Item 308074
8 rows fetched in 0.0004s (0.0042s) 
CTE Scan on rows  (cost=3188.19..3207.83 rows=982 width=230)
  CTE rows
    ->  Recursive Union  (cost=0.00..3188.19 rows=982 width=230)
          ->  Unique  (cost=0.00..15.46 rows=2 width=23)
                ->  Index Scan using ix_recursive_parent_orderer on t_recursive  (cost=0.00..15.45 rows=5 width=23)
                      Index Cond: (parent = 0)
          ->  Unique  (cost=313.84..314.33 rows=98 width=23)
                ->  Sort  (cost=313.84..314.08 rows=98 width=23)
                      Sort Key: c.parent, c.orderer
                      ->  Nested Loop  (cost=0.00..310.60 rows=98 width=23)
                            ->  WorkTable Scan on rows r  (cost=0.00..0.40 rows=20 width=4)
                            ->  Index Scan using ix_recursive_parent_orderer on t_recursive c  (cost=0.00..15.45 rows=5 width=23)
                                  Index Cond: (c.parent = r.id)

This gives us a single branch containing just the records we need: each one being the first child of its parent.

Now, what if we wanted to return a list of records, each being the second child to its parent?

Since each recursive part takes only one record as an input (and returns one record as an output), we can just replace DISTINCT ON (which returns the first child of each group) with a OFFSET 1 LIMIT 1:

WITH    RECURSIVE
        rows AS
        (
        SELECT  *
        FROM    (
                SELECT  *
                FROM    t_recursive
                WHERE   parent = 0
                ORDER BY
                        orderer
                OFFSET 1 LIMIT 1
                ) q
        UNION ALL
        SELECT  *
        FROM    (
                SELECT  c.*
                FROM    rows r
                JOIN    t_recursive c
                ON      c.parent = r.id
                ORDER BY
                        c.orderer
                OFFSET 1 LIMIT 1
                ) q2
        )
SELECT  *
FROM    rows
id parent orderer data
1 0 2540 Item 1
6 1 3405 Item 6
33 6 2884 Item 33
166 33 3084 Item 166
833 166 1848 Item 833
4169 833 993 Item 4169
20850 4169 3126 Item 20850
104251 20850 3021 Item 104251
521256 104251 5492 Item 521256
9 rows fetched in 0.0005s (0.0043s) 
CTE Scan on rows  (cost=1564.44..1564.66 rows=11 width=230)
  CTE rows
    ->  Recursive Union  (cost=3.09..1564.44 rows=11 width=230)
          ->  Limit  (cost=3.09..6.18 rows=1 width=23)
                ->  Index Scan using ix_recursive_parent_orderer on t_recursive  (cost=0.00..15.45 rows=5 width=23)
                      Index Cond: (parent = 0)
          ->  Limit  (cost=155.79..155.79 rows=1 width=23)
                ->  Sort  (cost=155.79..155.91 rows=49 width=23)
                      Sort Key: c.orderer
                      ->  Nested Loop  (cost=0.00..155.30 rows=49 width=23)
                            ->  WorkTable Scan on rows r  (cost=0.00..0.20 rows=10 width=4)
                            ->  Index Scan using ix_recursive_parent_orderer on t_recursive c  (cost=0.00..15.45 rows=5 width=23)
                                  Index Cond: (c.parent = r.id)

This query returns a whole branch of the items that are second children to their parents.

Both these queries are available in SQL Server 2005: despite the fact they do not distribute over UNION ALL, they can be rewritten using a ROW_NUMBER().

As was shown in the previous article, for some strange reason, SQL Server 2005 does not forbid this clause, but rather implements it in the incorrect way. This, however, allows writing the query we’re after.

Now, let’s make a query that would recursively return the first two children.

This requires a set-based solution, since first two children can come from different parents.

A grandchild that is third to its grandfather won’t count, even if it was a first child to the first child of the grandfather.

A first child to the third child of the grandfather won’t count either, even if it was the first grandchild to its grandfather.

To do this right, on each step the query should accept 2 items, return 2 items and process the items accepted at once.

This is also easy in a true set-based recursive CTE:

WITH    RECURSIVE
        rows AS
        (
        SELECT  *
        FROM    (
                SELECT  *
                FROM    t_recursive r
                WHERE   parent = 0
                ORDER BY
                        orderer
                LIMIT 2
                ) q
        UNION ALL
        SELECT  *
        FROM    (
                SELECT  c.*
                FROM    rows r
                JOIN    t_recursive c
                ON      c.parent = r.id
                ORDER BY
                        c.orderer
                LIMIT 2
                ) q
        )
SELECT  *
FROM    rows r
id parent orderer data
3 0 1686 Item 3
1 0 2540 Item 1
8 1 2181 Item 8
19 3 3370 Item 19
98 19 42 Item 98
99 19 1351 Item 99
497 99 1245 Item 497
496 99 1255 Item 496
2486 497 205 Item 2486
2484 496 362 Item 2484
12431 2486 29 Item 12431
12423 2484 311 Item 12423
62119 12423 1113 Item 62119
62120 12423 1121 Item 62120
310602 62120 341 Item 310602
310605 62120 468 Item 310605
16 rows fetched in 0.0007s (0.0043s) 
CTE Scan on rows r  (cost=3122.65..3123.09 rows=22 width=230)
  CTE rows
    ->  Recursive Union  (cost=0.00..3122.65 rows=22 width=230)
          ->  Limit  (cost=0.00..6.18 rows=2 width=23)
                ->  Index Scan using ix_recursive_parent_orderer on t_recursive r  (cost=0.00..15.45 rows=5 width=23)
                      Index Cond: (parent = 0)
          ->  Limit  (cost=311.58..311.58 rows=2 width=23)
                ->  Sort  (cost=311.58..311.82 rows=98 width=23)
                      Sort Key: c.orderer
                      ->  Nested Loop  (cost=0.00..310.60 rows=98 width=23)
                            ->  WorkTable Scan on rows r  (cost=0.00..0.40 rows=20 width=4)
                            ->  Index Scan using ix_recursive_parent_orderer on t_recursive c  (cost=0.00..15.45 rows=5 width=23)
                                  Index Cond: (c.parent = r.id)

This query takes all children of the previous set and returns the first two of their descendants.

We see that PostgreSQL, unlike SQL Server, implements the recursive CTE‘s in truly set-based way.

The recursive part of the query can accept a set on input, return a set on output and do the set-based operations on the set received.

]]> http://explainextended.com/2009/11/23/recursive-ctes-postgresql/feed/ 0 Shuffling rows: PostgreSQL http://explainextended.com/2009/10/06/shuffling-rows-postgresql/ http://explainextended.com/2009/10/06/shuffling-rows-postgresql/#comments Tue, 06 Oct 2009 19:00:08 +0000 Quassnoi http://explainextended.com/?p=3342 Answering questions asked on the site.

Josh asks:

I am building a music application and need to create a playlist of arbitrary length from the tracks stored in the database.

This playlist should be shuffled and a track can repeat only after at least 10 other tracks had been played.

Is it possible to do this with a single SQL query or I need to create a cursor?

This is in PostgreSQL 8.4

PostgreSQL 8.4 is a wise choice, since it introduces some new features that ease this task.

To do this we just need to keep a running set that would hold the previous 10 tracks so that we could filter on them.

PostgreSQL 8.4 supports recursive CTE‘s that allow iterating the resultsets, and arrays that can be easily used to keep the set of 10 latest tracks.

Here’s what we should do to build the playlist:

  1. We make a recursive CTE that would generate as many records as we need and just use LIMIT to limit the number
  2. The base part of the CTE is just a random record (fetched with ORDER BY RANDOM() LIMIT 1)
  3. The base part also defines the queue. This is an array which holds 10 latest records selected. It is initialized in the base part with the id of the random track just selected
  4. The recursive part of the CTE joins the previous record with the table, making sure that no record from the latest 10 will be selected on this step. To do this, we just use the array operator <@ (contained by)
  5. The recursive part adds newly selected record to the queue. The queue should be no more than 10 records long, that’s why we apply array slicing operator to it ([1:10])

Let’s create a sample table:

Table creation details

This table is quite simple: it just contains 1,000 tracks with generated names.

And here’s the query:

WITH    RECURSIVE
        shuffle AS
        (
        SELECT  *
        FROM    (
                SELECT  id, name, ARRAY[id] AS queue
                FROM    t_track
                ORDER BY
                        RANDOM()
                LIMIT 1
                ) q
        UNION ALL
        SELECT  *
        FROM    (
                SELECT  t.id, t.name, (t.id || s.queue)[1:10]
                FROM    shuffle s
                JOIN    t_track t
                ON      NOT ARRAY[t.id] <@ s.queue
                ORDER BY
                        RANDOM()
                LIMIT 1
                ) q
        )
SELECT  id, name, queue::VARCHAR
FROM    shuffle
LIMIT 30

View query details

This query selects first 30 records but the LIMIT clause can be changed to select an arbitrary number of records (including that exceeding 1,000), since we don’t apply any limits into the recursive part of the query.

Normally, the queue would be hidden but I left it just to illustrate what’s going on. As you can see, the queue holds the id‘s of last 10 records.

The query runs for 120 ms which is quite fast but could be yet improved using approaches described in PostgreSQL 8.4: sampling random rows. However, this will make the query too hard to read and ORDER BY RANDOM() is just fine to demonstrates the principle.

Hope that helps.

I’m always glad to answer the questions regarding database queries.

Ask me a question

]]> http://explainextended.com/2009/10/06/shuffling-rows-postgresql/feed/ 0 Adjacency list vs. nested sets: PostgreSQL http://explainextended.com/2009/09/24/adjacency-list-vs-nested-sets-postgresql/ http://explainextended.com/2009/09/24/adjacency-list-vs-nested-sets-postgresql/#comments Thu, 24 Sep 2009 19:00:35 +0000 Quassnoi http://explainextended.com/?p=3183 This series of articles is inspired by numerous questions asked on the site and on Stack Overflow.

What is better to store hierarchical data: nested sets model or adjacency list (parent-child) model?

First, let’s explain what all this means.

Adjacency list

Hierarchical relations (not to be confused with hierarchical data model) are 0-1:0-N transitive relations between entities of same domain.

For instance, ancestor-descendant relation is:

  • Transitive:
    • If A is an ancestor of B and B is an ancestor of C, then A is an ancestor of C
  • Irreflexive:
    • If A is an ancestor of B, then B is never an ancestor of A
  • 0-1:0-N
    • A can have zero, one or many children. A can have zero or one parents.

These relations can be represented by an ordered directed tree.

Tree is a simple directed graph (that with at most one directed edge between two different vertices) and relational model has means to represent simple graphs.

Two vertices are considered related (and therefore their primary keys forming a row in the table) if and only if they are connected with an edge.

This table along with the table defining the vertices identifies a graph completely by defining pairs of vertices connected by the edges. Each record in the table defines a pair of adjacent vertices, that’s why this representation is called adjacency list.

Adjacency lists can represent any simple directed graphs, not ony hierarchy trees. But due to the fact that this structure is most commonly used to define the parent-child relationships, the terms parent-child model and adjacency list model have almost become synonymous. However, they are not: adjacency list model is much wider and parent-child model is one of its implementations.

Now, since we have a tree here which implies 0-1:0-N relationship between the vertices, we can define the relation as a self-relation: the table defines both the entity and the relationship. Parent is just a one attribute among other attributes with a FOREIGN KEY reference to the table itself.

Since multple items can have no parents (and therefore be the roots of their trees), it’s sometimes useful to convert this tree into an arborescence: make a single fake root that considered a parent of all entries that have no actual parent.

This is a nice and elegant model, but until recently it had one drawback: it could not be used with SQL.

SQL, as we all know, deals with relational tables which can be transformed by the means of relational algebra.

It provides a way to do number of operations including relational multiplication, projection, sum etc.

However, earlier versions or SQL lacked recursion which is required to do certain operations efficiently. Namely, recursion is required to operate upon the adjacency list structure.

The most common operations are:

  • Find all descendants of a given node
  • Find all ancestors of a given node
  • Find all descendants of a given node up to a certain depth

The first two operations require recursion.

The third one does too if the depth level should serve as a parameter.

To work around this, the nested sets model was proposed by Michael Kamfonas and popularized by Joe Celko.

Nested sets

The idea of nested sets is quite simple and can be illustrated using one of the most popular ways to store hierarchical data, the XML.

How would we store the hierarchies in XML?

We would just use one tag to describe every item and nest it accordingly, like this:

<item id="0">
 <item id="1">
  <item id="2"/>
  <item id="3">
   <item id="4"/>
  </item>
  <item id="5">
   <item id="6">
    <item id="7"/>
   </item>
  </item>
 </item>
 <item id="8"/>
</item>

This is fine, but how to use this structure in a relational table?

As you can see, opening and closing tags of each node are contained on their own lines here. The opening and closing tags may (or may not) be the same for the nodes containing no children (this is not important).

We see that the node ranges never intersect: given two node ranges, their intersection always makes the range of either of the nodes or an empty set.

In other words, a node open within another node should also close within it.

If and only if this holds for each and every node, the nodes make a valid XML file and a valid hierarchy.

Now, the nested set model can be described in one sentence:

To store a hierarchy in a nested set model, we just store the line numbers of the opening and closing tags of each node as if it were an XML file.

Each set of nodes having a common ancestor is nested within the node of this ancestor. That’s why this model is called nested sets.

Historically, these line numbers are stored in columns named lft and rgt (since LEFT and RIGHT are reserved words in most SQL dialects).

That’s how this hierarchy would look in a nested sets model:

ID LFT RGT
0 1 14
1 2 12
2 3 3
3 4 6
4 5 5
5 7 11
6 8 10
7 9 9
8 13 13

This model is more SQL friendly, since the tasks described above can be performed in SQL without using recursion.

To find out all ancestors of a given node, we just select all nodes that contain its LFT boundary (which in a properly built hierarchy implies containing the RGT boundary too):

SELECT  hp.*
FROM    t_hierarchy hc
JOIN    t_hierarchy hp
ON      hc.lft BETWEEN hp.lft AND hp.rgt
WHERE   hc.id = ?

And to find the descendants, we shoud just reverse the condition, i. e. find all nodes that are contained between the current node’s boundaries:

SELECT  hc.*
FROM    t_hierarchy hp
JOIN    t_hierarchy hc
ON      hc.lft BETWEEN hp.lft AND hp.rgt
WHERE   hp.id = ?

Ironically, selecting all descendants up to a given depth (which is the least problem for the adjacency list as long as the depth is known in design time) is the hardest task for the nested set model. Even obtaning the list of immediate parents and immediate children is not so simple.

However, this is solvable. This is the query to get all descendants up to the third generation (that is node itself, all children and grandchildren):

SELECT  hc.*
FROM    t_hierarchy hp
JOIN    t_hierarchy hc
ON      hc.lft BETWEEN hp.lft AND hp.rgt
WHERE   hp.id = ?
        AND
        (
        SELECT  COUNT(*)
        FROM    t_hierarchy hn
        WHERE   hc.lft BETWEEN hn.lft AND hn.rgt
                AND hn.lft BETWEEN hp.lft AND hp.rgt
        ) <= 3

Unlike adjacency list model, the depth level can be parametrized in this query which makes it possible to use a single query for all depth level.

Nested set model can be relatively easily queried for, but it’s extremely hard to manage.

To insert a new child into a node or make an existing mode a child in adjacency list, everything we need is provide the new value of its parent column. With a single update we can move a whole branch.

To add a new node into a nested set model we should do exactly the same as if we were adding a new node into an XML file: all subsequent nodes are moved several lines further. Since the boundaries in SQL table represent the line numbers, we should do the same: calculate the offset and make a batch update to all nodes to the right of the updated or inserted one. Very hard to implement and very inefficient.

Now good news.

Three of four major systems (that is SQL Server, Oracle and PostgreSQL 8.4) now support recursion natively.

The fourth one (MySQL) does not support it, but it can be emulated to the extent required to run queries against the hierarchical data modelled according to adjacency list model:

In this series of articles, we will compare efficiency of the adjacency list model to that of the nested sets model.

PostgreSQL 8.4 is the system we begin with.

Analysis

PostgreSQL 8.4 supports recursive queries by means of such called hierarchical CTE’s.

A hierarchical CTE is an analog of this query:

SELECT  *
FROM    t_hierarchy h1
WHERE   … 

UNION ALL

SELECT  *
FROM    t_hierarchy h1
JOIN    t_hierarchy h2
ON    …
WHERE   … 

UNION ALL

SELECT  *
FROM    t_hierarchy h1
JOIN    t_hierarchy h2
ON    …
JOIN    t_hierarchy h3
ON    …
WHERE   …

…

with the theoretically unlimited number of UNION ALL‘s built at runtime and the results of each query cached and called recursively.

To define such a construct, one uses WITH RECURSIVE clause.

To compare both methods, we will create a sample table which combines both data models. Each node will have both parent and the boundaries (lft and rgt) defined. Then we will run the three most important queries, which, again, are:

  • Find all descendants of a given node
  • Find all ancestors of a given node
  • Find all descendants of a given node up to a certain depth

Here’s the script to create a sample table:

Table creation details

The table contains 8 levels of hierarchy with each node having 5 immediate children. This makes the table 2,441,405 records long.

Each record has a 100-byte long field stuffing which emulates the payload in actual tables.

The fields parent, lft and rgt are indexed.

All descendants

There are lots of descendants, that’s why we will select and aggregate the lengths of their stuffing fields. Since that field is not indexed, it will emulate selection of all values from an actual table rather well.

Nested sets

SELECT  SUM(LENGTH(hc.stuffing))
FROM    t_hierarchy hp
JOIN    t_hierarchy hc
ON      hc.lft BETWEEN hp.lft AND hp.rgt
WHERE   hp.id = 42

View query details

Nested sets is particularly good for this kind of query, since it requires a single range scan on the index on lft.

This query runs for 50 ms.

Adjacency list

WITH    RECURSIVE
        q AS
        (
        SELECT  id, stuffing
        FROM    t_hierarchy h
        WHERE   id = 42
        UNION ALL
        SELECT  hc.id, hc.stuffing
        FROM    q
        JOIN    t_hierarchy hc
        ON      hc.parent = q.id
        )
SELECT  SUM(LENGTH(stuffing))
FROM    q

View query details

This query is a trifle less efficient since it requires several index scans instead of a single one. However, the resulting range is of course the same (because the values returned are the same).

This query completes in 98 ms, or less than twice as long as the nested sets one.

All ancestors

Nested sets

SELECT  hp.id, hp.parent, hp.lft, hp.rgt, hp.data
FROM    t_hierarchy hc
JOIN    t_hierarchy hp
ON      hc.lft BETWEEN hp.lft AND hp.rgt
WHERE   hc.id = 1000000
ORDER BY
        hp.lft

View query details

This query returns much fewer rows than the previous one (only 9 rows instead of almost 200,000), but due to its nature it is much more slow and takes almost 2 seconds.

This is because we search the other way round in this case: instead of looking for indexed value within the range of constants, we need to search the constant against the list of ranges.

Ranges cannot be efficiently indexed using B-Tree indexes, that’s why PostgreSQL uses only part of the condition (hc.lft <= hp.rgt), builds a bitmap on it, scans the table using this bitmap and filters the values using the second part of the condition (hc.lft <= hp.rgt).

This is quite a costly operations since it requires an index scan (which PostgreSQL is not very good at) which returns almost half of all rows.

Adjacency list

WITH    RECURSIVE
        q AS
        (
        SELECT  h.*, 1 AS level
        FROM    t_hierarchy h
        WHERE   id = 1000000
        UNION ALL
        SELECT  hp.*, level + 1
        FROM    q
        JOIN    t_hierarchy hp
        ON      hp.id = q.parent
        )
SELECT  id, parent, lft, rgt, data
FROM    q
ORDER BY
        level DESC

View query details

Now this query is literally instant: only 4 ms which is within the time measurement error range.

Recursion does very good job here: since the hierarchy is limited, traversing the tree upwards takes only 9 index lookups on the PRIMARY KEY and then sorting of 9 values. Both operations are very simple and complete in no time.

Descendants up to a given level

Nested sets

We will run two queries: one with a node close to the root, the second one with a node far from the root.

SELECT  hc.id, hc.parent, hc.lft, hc.rgt, hc.data
FROM    t_hierarchy hp
JOIN    t_hierarchy hc
ON      hc.lft BETWEEN hp.lft AND hp.rgt
WHERE   hp.id = ?
        AND
        (
        SELECT  COUNT(*)
        FROM    t_hierarchy hn
        WHERE   hc.lft BETWEEN hn.lft AND hn.rgt
                AND hn.lft BETWEEN hp.lft AND hp.rgt
        ) <= 3

View query details for node 42

View query details for node 31,415

We see that the second query is reasonably fast (completes in 50 ms).

However, the first query (which is in fact more often used) takes 120.6 seconds, or more than 2 minutes!

This is because the query should count all ancestors for all descendants that are within the given node.

It’s fast for the nodes that are further from the root (since they don’t have lots of descendants), but it may become a real problem when trying to obtain, say, children and grandchildren of a root node.

And this is the task most online catalogs begin their work with: they need to show first-level categories and subcategories. 2 minutes is way too much for this.

Adjacency list

WITH    RECURSIVE
        q AS
        (
        SELECT  id, parent, lft, rgt, data, ARRAY[id] AS level
        FROM    t_hierarchy hc
        WHERE   id = ?
        UNION ALL
        SELECT  hc.id, hc.parent, hc.lft, hc.rgt, hc.data, q.level || hc.id
        FROM    q
        JOIN    t_hierarchy hc
        ON      hc.parent = q.id
        WHERE   array_upper(level, 1) < 3
        )
SELECT  id, parent, lft, rgt, data
FROM    q
ORDER BY
        level

Note the ORDER BY and level constructs. They are intended to preserve the tree-like ordering. Arrays are ordered lexicographically in PostgreSQL and each level contains the breadcrumbs from the root node to the current node.

Here are the query results:

View query details for node 42

View query details for node 31,415

As we can see, both queries complete in a little more than 5 ms (instantly) and this time does not depend on the proximity to the root node.

Summary

We have compared the three most common queries that are usually issued against the hierarachical data:

  1. Find all descendants of a given node
  2. Find all ancestors of a given node
  3. Find all descendants of a given node up to a certain depth

The nested set model the fastest for the first query (0.05 s), however, adjacency list shows good performance and is very fast too (selecting 200,000 rows is a matter of less than 0.1 second).

For the second query, adjacency list is much faster, however, the nested sets are still usable.

Finally, for the third query, nested sets model shows dependency on the node.

For a node that has few descendants, the query is rather fast, however, for a node close to the root (to say nothing of the root itself) this query is intolerably slow.

Adjacency list shows superb performance on both nodes.

Conclusion

Given the said above and taking into account that the nested sets model is much harder to manage, we can conclude that adjacency list model should be used to manage hierarchical data in PostgreSQL 8.4.

Nested sets model was a very smart invention to manage hierarchical data in an environment that allowed no recursion. But now, when recursive queries are finally available, adjacency list model is just better.

It yields excellent performance on all three types of queries, outperforms nested sets in two of three most used queries and is extremely simple to manage.

To be continued.

]]> http://explainextended.com/2009/09/24/adjacency-list-vs-nested-sets-postgresql/feed/ 0