EXPLAIN EXTENDED » MySQL http://explainextended.com How to create fast database queries Tue, 09 Mar 2010 18:43:36 +0000 http://wordpress.org/?v=2.9.2 en hourly 1 Aggregates and LEFT JOIN http://explainextended.com/2010/03/05/aggregates-and-left-join/ http://explainextended.com/2010/03/05/aggregates-and-left-join/#comments Fri, 05 Mar 2010 20:00:09 +0000 Quassnoi http://explainextended.com/?p=4548 From Stack Overflow:

I have a table product with products and table sale with all sale operations that were done on these products.

I would like to get 10 most often sold products today and what I did is this:

SELECT  p.*, COUNT(s.id) AS sumsell
FROM    product p
LEFT JOIN
        sale s
ON      s.product_id = p.id
        AND s.dt >= '2010-01-01'
        AND s.dt < '2010-01-02'
GROUP BY
        p.id
ORDER BY
        sumsell DESC
LIMIT 10

, but performance of it is very slow.

What can I do to increase performance of this particular query?

The query involves a LEFT JOIN which in MySQL world means that products will be made leading in the query. Each record of product will be taken and checked against sale table to find out the number of matching records. If no matching records are found, 0 is returned.

Let’s create the sample tables:

Table creation details

The table contains 500,000 products and 500,000 random sales (1,440 sales per day).

Now, let’s run the query similar to the author’s one. I adjusted the period so that fewer than 10 actual sales were made during the period and LEFT JOIN records can be seen in the table:

SELECT  p.*, COUNT(s.id) AS sumsell
FROM    product p
LEFT JOIN
        sale s
ON      s.product_id = p.id
        AND s.dt >= '2010-01-01'
        AND s.dt < '2010-01-01 00:07:00'
GROUP BY
        p.id
ORDER BY
        sumsell DESC, p.id
LIMIT 10

View query details

The query runs for 3 seconds.

We see that, first, product is made leading, and, second, only a part of the index on sale (product, dt) is used: each sale is only filtered on product, not on date.

Since there were only 7 sales during the period we have chosen, it would be a wise decision to make sale leading in the join so that it could be filtered on date and the resulting recordset was then joined to product. This would result in at most 7 PRIMARY KEY seeks instead of 500,000 range scans and would be much more efficient.

However, this is only possible with the INNER JOIN, and if there are less then 10 products sold within the time period, we will not see the rest.

To work around this, we need to emulate the LEFT JOIN:

  1. Find the products sold within the time period, using an INNER JOIN of product with the resultset containg aggregated sales.

  2. Find the products not sold within the time period, using NOT EXISTS predicate.

  3. Concatenate the two resultsets using UNION ALL.

The step 2 implies that product is leading again, so normally it would not be much of improvement. But in our case, we don’t need the whole recordset, we only need the top 10 sales.

So we can just order and limit the recordsets retrieved on steps 1 and 2 to ten records each, concatenate them, then order and limit the resulting recordset again to ten records.

The second resultset will contain a hardcoded 0 in the sumsell, so we just need to order it on product.id. Since product is an InnoDB table and product.id is a clustered PRIMARY KEY, this is not a problem.

Here’s the query:

SELECT  p.*, sumsell
FROM    (
        SELECT  *
        FROM    (
                SELECT  product_id, sumsell
                FROM    (
                        SELECT  product_id, COUNT(*) AS sumsell
                        FROM    sale si
                        WHERE   dt >= '2010-01-01'
                                AND dt < '2010-01-01 00:07:00'
                        GROUP BY
                                product_id
                        ) si
                ORDER BY
                        sumsell DESC, product_id
                LIMIT 10
                ) q1
        UNION ALL
        SELECT  *
        FROM    (
                SELECT  p.id, 0
                FROM    product p
                WHERE   NOT EXISTS
                        (
                        SELECT  NULL
                        FROM    sale si
                        WHERE   product_id = p.id
                                AND dt >= '2010-01-01'
                                AND dt < '2010-01-01 00:07:00'
                        )
                ORDER BY
                        p.id
                LIMIT 10
                ) q2
        ORDER BY
                sumsell DESC, product_id
        LIMIT 10
        ) q
JOIN    product p
ON      p.id = q.product_id

View query details

This query completes in less than 7 ms (which is comparable to the time measurement error).

]]> http://explainextended.com/2010/03/05/aggregates-and-left-join/feed/ 0 Matching sets: aggregates vs. first miss http://explainextended.com/2010/02/25/matching-sets-aggregates-vs-first-miss/ http://explainextended.com/2010/02/25/matching-sets-aggregates-vs-first-miss/#comments Thu, 25 Feb 2010 20:00:47 +0000 Quassnoi http://explainextended.com/?p=4436 From Stack Overflow:

Here is my schema:

Suppliers
sid sname address
Parts
pid pname color
Catalog
sid pid cost

I need to find the sids of suppliers who supply every red part or every green part.

This task requires matching the sets.

We need to compare two sets here: the first one is the set of the parts of given color; the second one is the set of parts provided by a given supplier. The former should be the subset of the latter.

Unlike other engines, MySQL does not provide the set operators like EXCEPT or MINUS that allow to check the subset / superset relationship very easily. We have to use the record-based solutions.

There are two ways to check that:

  • First miss technique: test each record from the subset candidate against the superset candidate, returning FALSE if there is no match.
  • Aggregate technique: compare the number of records in the subset candidate to the number of records in their intersection. If the numbers are equal, the sets match

Let’s test which way is faster in which cases. To do this, we will need some sample tables:

Table creation details

There are 10,000 suppliers and 2,000 parts.

The parts can be red, green or blue. As for red and green, there are 10 parts of each color, and they are distributed evenly across the suppliers. With blue, the situation is different: there are 1980 blue parts and only 200 first suppliers provide it. However, for each of the blue part suppliers, the probability of each blue part to be available is very high.

First miss

The first miss is a combination of NOT IN / NOT EXISTS clauses that immediately return FALSE whenever a single non-matching record is found. Since MySQL can only do nested loops, the performance of these queries is heavily dependent on proper indexing.

Let’s run this query to search for red or green parts:

SELECT  *
FROM    suppliers s
WHERE   EXISTS
        (
        SELECT  NULL
        FROM    (
                SELECT  'red' AS color
                UNION ALL
                SELECT  'green' AS color
                ) ci
        WHERE   color NOT IN
                (
                SELECT  color
                FROM    parts p
                WHERE   p.pid NOT IN
                        (
                        SELECT  pid
                        FROM    catalog c
                        WHERE   c.sid = s.sid
                        )
                )
        )

View query details

This query runs for 2.17 s.

The same query for the blue parts:

SELECT  *
FROM    suppliers s
WHERE   EXISTS
        (
        SELECT  NULL
        FROM    (
                SELECT  'blue' AS color
                ) ci
        WHERE   color NOT IN
                (
                SELECT  color
                FROM    parts p
                WHERE   p.pid NOT IN
                        (
                        SELECT  pid
                        FROM    catalog c
                        WHERE   c.sid = s.sid
                        )
                )
        )

View query details

This time the query is a little bit slower: 2.43 s.

Aggregate

In the aggregate solution, we first calculate the total number of parts for each color and the count the parts of this color supplied by each supplier. If the counts match, the sets match too.

Let’s try it on red and green parts first:

SELECT  c.sid, p.color
FROM    (
        SELECT  color, COUNT(*) AS total
        FROM    parts
        WHERE   color IN ('red', 'green')
        GROUP BY
                color
        ) t
JOIN    parts p
ON      p.color = t.color
JOIN    catalog c
ON      c.pid = p.pid
GROUP BY
        sid, color, total
HAVING  COUNT(*) = total

View query details

And the same query on the blue parts:

SELECT  c.sid, p.color
FROM    (
        SELECT  color, COUNT(*) AS total
        FROM    parts
        WHERE   color IN ('blue')
        GROUP BY
                color
        ) t
JOIN    parts p
ON      p.color = t.color
JOIN    catalog c
ON      c.pid = p.pid
GROUP BY
        sid, color, total
HAVING  COUNT(*) = total

View query details

We see that the queries complete in 1.14 seconds and 3.09 seconds, accordingly.

The aggregate method is more efficient for the red and green parts, while the first miss is more efficient for the blue parts.

Analysis

What is the reason of such a difference in performance?

The first miss method generally needs to parse less records (only those before the first miss), but each record is searched for in a nested loop, starting from the index root. The aggregate method needs to parse all records to calculate the COUNT(*), but these records are fetched in a sequential index access.

The red and green parts have the large number of small sets, with at most 10 records in each. The probability of the miss is relatively small: a large number of records should be parsed before any record is missed in a set. The aggregates, on the other hand, can be calculated very fast, since there are relatively few records to aggregate.

With the blue parts, the situation is different. There are few large sets, and calculating the aggregates requires fetching, sorting and grouping of too many records. First misses, on the other hand, occurs almost instantly: the vast majority of the suppliers do not offer any blue parts at all.

Summary

As it often happens, the performance of the two methods to compare the sets depends on the data distribution.

The sets with less records and lower probability of the record miss will benefit from the aggregate method, since the performance increase caused by the sequential access to the records overweights the need to parse a larger number of the records.

The sets with more records and higher probability of a miss will return the misses very soon, so the first miss method is more beneficial for them.

]]> http://explainextended.com/2010/02/25/matching-sets-aggregates-vs-first-miss/feed/ 0 Tags in nested sets: efficient indexing http://explainextended.com/2010/02/16/tags-in-nested-sets-efficient-indexing/ http://explainextended.com/2010/02/16/tags-in-nested-sets-efficient-indexing/#comments Tue, 16 Feb 2010 20:00:11 +0000 Quassnoi http://explainextended.com/?p=4326 Answering questions asked on the site.

Travis asks:

I just read your excellent post: Hierarchical data in MySQL: parents and children in one query.

I am currently trying to do something very similar to this. The main difference is, I am working with data that is in a nested set.

I would like to construct a query that will return a resultset of nodes matched with LIKE, the ancestors of each of these nodes, and the immediate children of each of these nodes.

This is a very interesting question: it allows to demonstrate the usage of three types of indexes usable by MyISAM tables: BTREE, SPATIAL and FULLTEXT.

Nested sets is one of two models that are most often used to store hierarchical data in a relational table (the other one being adjacency list model). Unlike adjacency list, nested sets does not require ability to write recursive queries, which SQL originally lacked and still lacks now in MySQL (albeit it can be emulated to some extent). It is widely used in MySQL world.

I described both methods and their comparison in the article I wrote some time ago:

The main problem with the nested sets model is that though it is extremely fast with selecting descendants of a given node, it is very slow in selecting ancestors.

This is because of the way the B-Tree indexes work. They can be used to query for values of a column within the range of two constants, but not for the values of two columns holding a single constant between them. One needs the first condition to select the children (found between the lft and rgt of the given node), and the second condition to select the ancestor (with lft and rgt containing the lft and rgt of the given node).

That’s why selecting the children is fast and selecting the ancestors is slow.

To work around this, the sets that form the hierarchy can be described as geometrical objects, with the larger sets containing the smaller sets. These sets can be indexed with a SPATIAL index which is designed specially for this purpose and both children and ancestors can be queried for very efficiently.

Unfortunately, finding the depth level is quite a hard task for the nested sets model even with the SPATIAL indexes.

It would be quite an easy task is MySQL supported recursion: we could just run a query to find the siblings of each record by skipping their whole domains recursively.

However, MySQL’s recursion support is very limited and it relies on the session variables, which are not recommended to use in the complex queries.

To cope with this, we need to mix the nested sets and the adjacency list models. Hierarchy will be stored in two seemingly redundant ways: the unique parent and the LineString representing the nested sets.

This will help us to use the R-Tree index to find all ancestors of a given node and also use B-Tree index to find its immediate children.

Finally, the question mentions using LIKE to find the initial nodes. LIKE predicate with the leading wildcards is not sargable in MySQL. However, it seems that the leading wildcards are only used to split the words. In this case, a FULLTEXT index and the MATCH query would be much more efficient, since FULLTEXT index allows indexing a single record with several keys (each one corresponding to a single word in the column’s text), so a search for the word in the space separated or a comma separated list uses the index and is much faster than scanning the whole table.

Hence, the query would use all three main types of indexes: BTREE, SPATIAL and FULLTEXT.

To illustrate everything said above, let’s create a sample table:

Table creation details

This table represents a 7 level hierarchy, with 5 children to each non-leaf item and combines nested sets and adjacency list models.

The sets are stored in the plain lft and rgt columns as well as in the combined column of type LineString which represents a diagonal of a box horizontally spanning the interval from lft to rgt.

Selecting nodes using FULLTEXT

First, let’s select the nodes tagged with a given tag.

RLIKE can be used for that but it is not very efficient:

SELECT  id, tags
FROM    t_hierarchy
WHERE   tags RLIKE '[[:<:]]tag13480[[:>:]]'

View query results

This query uses full table scan and runs for 1.25 second.

To improve the query, we should rewrite the WHERE condition using MATCH predicate (which in its turn allows a FULLTEXT index to be used for the search):

SELECT  id, tags
FROM    t_hierarchy
WHERE   MATCH(tags) AGAINST('+tag13480' IN BOOLEAN MODE)
id tags
3 tag13480 tag33087 tag124996
440166 tag248489 tag271789 tag13480
212847 tag104605 tag13480 tag53585
161378 tag181040 tag231320 tag13480
324394 tag13480 tag181947 tag269297
400074 tag13480 tag176642 tag242772
6 rows fetched in 0.0005s (0.0043s)
id select_type table type possible_keys key key_len ref rows filtered Extra
1 SIMPLE t_hierarchy fulltext fx_hierarchy_tags fx_hierarchy_tags 0 1 100.00 Using where 
select `20100216_index`.`t_hierarchy`.`id` AS `id`,`20100216_index`.`t_hierarchy`.`tags` AS `tags` from `20100216_index`.`t_hierarchy` where (match `20100216_index`.`t_hierarchy`.`tags` against ('+tag13480' in boolean mode))

This query returns the same results but does it much faster, in only 4 ms.

Selecting ancestors using SPATIAL

Now, when we have a list of nodes, we should build the list of ancestors for each node.

Since our model combines adjacency list and nested sets, it is possible to use either representation to build a query. However, the adjacency list model requires recursion, and, while it is possible to emulate it, it only works for a single-node query.

With nested sets, selecting the list of ancestors is much more simple: we should just selecting all records whose sets contains the sets of the node. This can be done using MBRContains (which is capable of using the SPATIAL index).

The query, however, will return us the ancestors in a plain list. To find out the level of each ancestor, we should put some more effort. Since the sets are nested and the lft and rgt fields naturally maintain the hierarchical order, it is enough just to enumerate the ancestors in that order. It would be very simple to do — if only MySQL supported ROW_NUMBER(). But it doesn’t, of course, so to enumerate the ancestors we should self-join them and just count the number of each ancestor’s ancestors:

SELECT  hc.id, ha.id, COUNT(*) AS cnt
FROM    t_hierarchy hc
STRAIGHT_JOIN
        t_hierarchy ha
ON      MBRContains(ha.sets, hc.sets)
STRAIGHT_JOIN
        t_hierarchy hcnt
ON      MBRContains(hcnt.sets, ha.sets)
WHERE   MATCH(hc.tags) AGAINST('+tag13480' IN BOOLEAN MODE)
GROUP BY
        hc.id, ha.id
ORDER BY
        hc.id, cnt

View query results

The algorithm that calculates the level of each ancestor is not very efficient, however, it does its job quite well, and the query completes in only 20 ms.

Selecting immediate children using BTREE

Selecting the immediate children is the easiest task: we just need an equijoin on parent. The level of each of the node’s children will be that of the node plus 1, so calculating it is quite simple too:

SELECT  h.id, hc.id, cnt + 1
FROM    (
        SELECT  hn.id, COUNT(*) AS cnt
        FROM    t_hierarchy hn
        STRAIGHT_JOIN
                t_hierarchy hcnt
        ON      MBRContains(hcnt.sets, hn.sets)
        WHERE   MATCH(hn.tags) AGAINST('+tag13480' IN BOOLEAN MODE)
        GROUP BY
                hn.id
        ) h
JOIN    t_hierarchy hc
ON      hc.parent = h.id

View query results

Putting it together

Now we should just combine the two queries and apply nice formatting to them:

SELECT  h.id,
        CONCAT(LPAD('', (level - 1) * 2, ' '), h.data) AS name,
        CASE WHEN hq.id = hq.node THEN '*' ELSE '' END AS hit
FROM    (
        SELECT  hc.id AS node, ha.id AS id, COUNT(*) AS level
        FROM    t_hierarchy hc
        STRAIGHT_JOIN
                t_hierarchy ha
        ON      MBRContains(ha.sets, hc.sets)
        STRAIGHT_JOIN
                t_hierarchy hcnt
        ON      MBRContains(hcnt.sets, ha.sets)
        WHERE   MATCH(hc.tags) AGAINST('+tag13480' IN BOOLEAN MODE)
        GROUP BY
                hc.id, ha.id
        UNION ALL
        SELECT  h.id, hc.id, cnt + 1 AS level
        FROM    (
                SELECT  hn.id, COUNT(*) AS cnt
                FROM    t_hierarchy hn
                STRAIGHT_JOIN
                        t_hierarchy hcnt
                ON      MBRContains(hcnt.sets, hn.sets)
                WHERE   MATCH(hn.tags) AGAINST('+tag13480' IN BOOLEAN MODE)
                GROUP BY
                        hn.id
                ) h
        JOIN    t_hierarchy hc
        ON      hc.parent = h.id
        ) hq
JOIN    t_hierarchy h
ON      h.id = hq.id
ORDER BY
        node, level, lft
id name hit
3 Value 3 *
16 Value 16
17 Value 17
18 Value 18
19 Value 19
20 Value 20
1 Value 1
10 Value 10
51 Value 51
257 Value 257
1290 Value 1290
6454 Value 6454
32275 Value 32275
161378 Value 161378 *
2 Value 2
13 Value 13
67 Value 67
340 Value 340
1702 Value 1702
8513 Value 8513
42569 Value 42569
212847 Value 212847 *
3 Value 3
20 Value 20
103 Value 103
518 Value 518
2594 Value 2594
12975 Value 12975
64878 Value 64878
324394 Value 324394 *
4 Value 4
25 Value 25
127 Value 127
639 Value 639
3200 Value 3200
16002 Value 16002
80014 Value 80014
400074 Value 400074 *
5 Value 5
27 Value 27
140 Value 140
704 Value 704
3521 Value 3521
17606 Value 17606
88033 Value 88033
440166 Value 440166 *
46 rows fetched in 0.0039s (0.0466s)
id select_type table type possible_keys key key_len ref rows filtered Extra
1 PRIMARY <derived2> ALL 46 100.00 Using temporary; Using filesort
1 PRIMARY h eq_ref PRIMARY PRIMARY 4 hq.id 1 100.00
2 DERIVED hc fulltext sx_hierarchy_sets,fx_hierarchy_tags fx_hierarchy_tags 0 1 100.00 Using where; Using temporary; Using filesort
2 DERIVED ha range sx_hierarchy_sets sx_hierarchy_sets 34 1 48828000.00 Range checked for each record (index map: 0×10)
2 DERIVED hcnt range sx_hierarchy_sets sx_hierarchy_sets 34 1 48828000.00 Range checked for each record (index map: 0×10)
3 UNION <derived4> ALL 6 100.00
3 UNION hc ref ix_hierarchy_parent ix_hierarchy_parent 4 h.id 9207 100.01
4 DERIVED hn fulltext sx_hierarchy_sets,fx_hierarchy_tags fx_hierarchy_tags 0 1 100.00 Using where; Using temporary; Using filesort
4 DERIVED hcnt range sx_hierarchy_sets sx_hierarchy_sets 34 1 48828000.00 Range checked for each record (index map: 0×10)
UNION RESULT <union2,3> ALL 
select `20100216_index`.`h`.`id` AS `id`,concat(convert(lpad('',((`hq`.`level` - 1) * 2),' ') using utf8),`20100216_index`.`h`.`data`) AS `name`,(case when (`hq`.`id` = `hq`.`node`) then '*' else '' end) AS `hit` from (select `20100216_index`.`hc`.`id` AS `node`,`20100216_index`.`ha`.`id` AS `id`,count(0) AS `level` from `20100216_index`.`t_hierarchy` `hc` straight_join `20100216_index`.`t_hierarchy` `ha` straight_join `20100216_index`.`t_hierarchy` `hcnt` where (match `20100216_index`.`hc`.`tags` against ('+tag13480' in boolean mode)) group by `20100216_index`.`hc`.`id`,`20100216_index`.`ha`.`id` union all select `h`.`id` AS `id`,`20100216_index`.`hc`.`id` AS `id`,(`h`.`cnt` + 1) AS `level` from (select `20100216_index`.`hn`.`id` AS `id`,count(0) AS `cnt` from `20100216_index`.`t_hierarchy` `hn` straight_join `20100216_index`.`t_hierarchy` `hcnt` where ((match `20100216_index`.`hn`.`tags` against ('+tag13480' in boolean mode)) and contains(`20100216_index`.`hcnt`.`sets`,`20100216_index`.`hn`.`sets`)) group by `20100216_index`.`hn`.`id`) `h` join `20100216_index`.`t_hierarchy` `hc`) `hq` join `20100216_index`.`t_hierarchy` `h` where (`20100216_index`.`h`.`id` = `hq`.`id`) order by `hq`.`node`,`hq`.`level`,`20100216_index`.`h`.`lft`

For each matching node, the query returns the node itself, its ancestors and its immediate children. The matching nodes are marked with the asterisk in the field hit.

The query efficiently combines the FULLTEXT, SPATIAL and BTREE indexes and completes in only 40 ms.

Hope that helps.

I’m always glad to answer the questions regarding database queries.

Ask me a question

]]> http://explainextended.com/2010/02/16/tags-in-nested-sets-efficient-indexing/feed/ 0 Index search time depends on the value being searched http://explainextended.com/2010/02/04/index-search-time-depends-on-the-value-being-searched/ http://explainextended.com/2010/02/04/index-search-time-depends-on-the-value-being-searched/#comments Thu, 04 Feb 2010 20:00:58 +0000 Quassnoi http://explainextended.com/?p=4183 Answering questions asked on the site.

Daniel asks:

I have a table which stores track titles in a VARCHAR(200) field. The field is indexed, but searching for titles beginning with a letter Z is noticeably slower than for those beginning with A, and the closer the initial letter is to Z, the slower is the query.

My understanding is that a full table scan occurs, but EXPLAIN shows that the index is used. Besides, the table is quite large but the query is still reasonably fast.

Could you please explain this behavior?

MySQL stores its indexes in B-Tree data structures.

The Wikipedia link above explains the structure quite well so I won’t repeat it. I’ll rather draw a picture similar to the one in the article:

This picture is quite self-explanatory. The records are sorted in a tree order, so if you are searching for a certain value, say, 11, you, starting from the first page, should find the link to follow. To do this, you need to find the pair of values less than and greater than 11. In this case, you should follow the link which is between 8 and 12. Then you search for the next pair, etc, until you find your value or reach the end and make sure that your value is not there.

Following the links is quite simple, but how does the engine search for the values within one page?

This depends on how you declared the table.

MyISAM supports two algorithms for storing the index keys in a page: packed keys and unpacked keys.

Unpacked keys are what you are seeing on the picture above: each page just stores the key values and the links to the pages down the tree. This is very simple.

Packed keys are designed to improve performance on character data. Many words and phrases, especially those that are close to each other, start with the same sequence of characters.

If you are going to store track names like:

The Man Who Sold The World
The Man Who Invented Himself
The Man Who Has Everything

, MyISAM can optimize it in terms of storage space and store them like this

(The Man Who) Sold The World
(×11) Invented Himself
(×11) Has Everything

This is called key compression: instead of repeating the key value for each record, MyISAM just stores the longest common prefix once and prepends the subsequent records that share it with its lengths. This makes the keys shorter and the index more compact.

However, this affects the index search time.

With an unpacked index, a binary search is applied to find the keys within each level page.

With a packed index, this won’t work: you need to know the value of two keys to compare them and not every record contains full information about the key.

So in case of a packed index, MySQL remembers the value of the prefix and iterates the records one by one. This is less efficient than a binary search, but due to the fact that much more records can fit on on page, this keeps the amount of page traversals to a minimum and overall efficiency increases.

But the records on one page still need to be compared and searched for. And with a linear search, the keys with less values tend to require less iterations than those with greater values.

Let’s look on the picture above again. The keys are searched left to right.

To search for key 1, we only need two comparisons: compare to 4, get the next page, compare to 1.

But to search for key 15, we need to compare to 4, 8, 12 then get to the next page and compare to 13, 14 and finally to 15.

This is 6 operations compared to 2 required to fetch the first key.

Now, let’s create the sample tables and see some figures:

Table creation details

There are two MyISAM tables with randomly generated character sequences like this:

id name
1 RTUPPH
2 RKZQJW
3 FKMEKL
4 BYOZFE
5 GSTRAF
6 YBNMSG
7 ZEZKCE
8 PMPNUJ
9 OQMMYH
10 OYAFDZ
49999 DZMKRC
50000 NHYWLR

The structure of the tables is identical, except that t_packed packs keys and t_unpacked does not.

Unpacked keys

SELECT  COUNT(*)
FROM    t_source s
LEFT JOIN
        t_unpacked p
ON      p.name = IF(s.id > -1, _UTF8'ABCDEF', NULL)
COUNT(*)
50000
1 row fetched in 0.0001s (0.7656s) 
SELECT  COUNT(*)
FROM    t_source s
LEFT JOIN
        t_unpacked p
ON      p.name = IF(s.id > -1, _UTF8'GHIJKL', NULL)
COUNT(*)
50000
1 row fetched in 0.0001s (0.7331s) 
SELECT  COUNT(*)
FROM    t_source s
LEFT JOIN
        t_unpacked p
ON      p.name = IF(s.id > -1, _UTF8'NOPQRS', NULL)
COUNT(*)
50000
1 row fetched in 0.0001s (0.7656s) 
SELECT  COUNT(*)
FROM    t_source s
LEFT JOIN
        t_unpacked p
ON      p.name = IF(s.id > -1, _UTF8'ZYXWVU', NULL)
COUNT(*)
50000
1 row fetched in 0.0001s (0.7712s)

The queries against the strings beginning with A, G, N and Z take the same time. All queries have been run several times to populate the cache and the execution times are consistent.

The LEFT JOIN against a non-existent value was used in the query to avoid stopping on a found key and make the query traverse the index as much as possible. We also put a formal dependency on s.id here so that t_source is always leading in the join and no const optimizations are performed.

Packed keys

Let’s try the same queries on packed keys:

SELECT  COUNT(*)
FROM    t_source s
LEFT JOIN
        t_packed p
ON      p.name = IF(s.id > -1, _UTF8'ABCDEF', NULL)
COUNT(*)
50000
1 row fetched in 0.0001s (1.9531s) 
SELECT  COUNT(*)
FROM    t_source s
LEFT JOIN
        t_packed p
ON      p.name = IF(s.id > -1, _UTF8'GHIJKL', NULL)
COUNT(*)
50000
1 row fetched in 0.0001s (2.3593s) 
SELECT  COUNT(*)
FROM    t_source s
LEFT JOIN
        t_packed p
ON      p.name = IF(s.id > -1, _UTF8'NOPQRS', NULL)
COUNT(*)
50000
1 row fetched in 0.0002s (2.7812s) 
SELECT  COUNT(*)
FROM    t_source s
LEFT JOIN
        t_packed p
ON      p.name = IF(s.id > -1, _UTF8'ZYXWVU', NULL)
COUNT(*)
50000
1 row fetched in 0.0002s (2.9375s)

The query against the string beginning with the letter Z takes more than 50% more time than the query against a string beginning with A.

Summary

Here’s a little summary table:

Search string Unpacked key Time, % Packed key Time, %
ABCDEF 0.7656 100.00 1.9531 100.00
GHIJKL 0.7331 95.75 2.3593 120.79
NOPQRS 0.7656 100.00 2.7812 142.39
ZYXWVU 0.7712 100.73 2.9375 150.40

We see that the value being searched for does not affect time to search the index with unpacked keys but seriously affects performance of the indexes with packed keys.

This increase is due to linear search used to locate the records within a single page.

Note that not any pair of records in a page (both with packed and unpacked keys) have a corresponding lower-level page containing intermediate values. There can be leaves and branches on the same depth in the tree.

This can lead to some artifacts: certain strings can be found (or proved absent) faster than the others. However, in average, all records have same depth. And in average, with packed indexes, the need for linear search increases the time required to find the keys with the greater values.

Hope that helps.

I’m always glad to answer the questions regarding database queries.

Ask me a question

]]> http://explainextended.com/2010/02/04/index-search-time-depends-on-the-value-being-searched/feed/ 0 Join on overlapping date ranges http://explainextended.com/2010/02/01/join-on-overlapping-date-ranges/ http://explainextended.com/2010/02/01/join-on-overlapping-date-ranges/#comments Mon, 01 Feb 2010 20:00:02 +0000 Quassnoi http://explainextended.com/?p=4050 This post is inspired by a discussion with John Didion:

Is there any way to optimize the query for overlapping ranges in MySQL if both ranges are dynamic?

I have two tables, each with integer range columns (specified as LineString), and I want to find rows that overlap.

No matter what I try, the query planner never uses any indexes.

This question addresses a well-known problem of efficient searching for the intersecting intervals. The queries that deal with it require ability to search for the intervals (stored in two distinct columns) containing a constant scalar value.

Plain B-Tree indexes used by most databases do not speed up the queries like that. However, MySQL supports SPATIAL indexes that can index two-dimensional shapes and efficiently search for the shapes containing a given point.

With a little effort, time intervals can be converted into the geometrical objects, indexed with a SPATIAL index and searched for the given point in time (also presented as a gemetrical object). This is described in the article about overlapping ranges in MySQL.

The query described in that article, however, searches for the intervals overlapping a constant range, provided as a parameter to the query. Now, I will discuss how to adapt the query for a JOIN between two tables.

Let’s create two sample tables, each containing a set of time ranges stored as geometrical objects, and find all records from both tables whose ranges overlap:

Table creation details

There are two tables, t_big and t_small.

t_big contains 120,000 records with intervals starting each hour and having random length from 0 to 2 hours, 1 hour in average.

t_small contains 5,000 records with intervals starting each day and having random length from 0 to 2 days, 1 day in average.

The intervals are presented as the LineString records with a single line connecting Point(-1, range_start) and Point(1, range_end). range_start and range_end are presented as the UNIX timestamps (numbers of seconds since Jan 1st, 1970). To avoid year 2038 problem, we use TIMESTAMPDIFF rather than UNIX_TIMESTAMP to calculate the value.

Each table is indexes with a SPATIAL index on the rg field.

First of all, we need to make sure that the index is created well.

MySQL uses R-Tree for the SPATIAL indexes, which works according to the principle of least enlargement to the bounding box when adding a value to the index. Each leaf node is covered by a certain minimum bounding box stored in the branch node, and the branch which needs to be increased to the least extent is chosen as a new parent for the element.

Our ranges are one-dimensional, but MySQL requires the spatial data to be two-dimensional. That’s why we need to use either of the dimensions to represent the time ranges and fill another one to span some constant range so that the box areas would be proportional to the range lengths.

However, if we just fill both ends of the LineString with a constant, this will make the lines pure horizontal or pure vertical. Since they all reside on one line, the minimal bounding boxes of any set of the LineString’s will have zero area (since they all will be of zero height). MySQL will not be able to choose the least enlargement: in a two-dimensional space, any horizontal enlargement of a one-dimensional line will be a zero. MySQL will just append the entry to any randomly chosen branch in no certain order. This makes the index very unbalanced and in fact unusable.

To make the index usable, we need to make the boxes have positive area so that the least enlargement principle could do its job. That’s why we create the LineString fields diagonally, with the first coordinates being -1 for the start point and 1 for the end point.

Now, we should make a query that would find the intersections of these ranges.

MySQL documentation evasively mentions that the predicates supported by the spatial indexes are MBRContains and MBRWithin (with the latter being just the synonym for MBRContains with the argument order reversed), with a side note that in future releases, spatial indexes may also be used for optimizing other functions

If we were to comply with the documentation and limit ourselves to using only these two functions for the spatial indexes, the query would still be possible and quite efficient.

MBRContains checks if the bounding box of the second argument lies within the bounding box for the first argument. This is not always true for the overlapping ranges. If the ranges overlap partially, that is the first range both starts and ends earlier than the second range, then no ranges lie completely within each other’s bounds, and MBRContains will return 0 for any argument order.

However, there is a simple condition that checks whether the ranges do overlap.

With any pair of ranges, there is a range that starts later (or at the same time) than another. If the ranges overlap, then the start of that range should lie within the bounds of its counterpart. Indeed, the start of the latest range should be equal to or greater then the start of the earliest range (by definition) and equal to of less than the end of the earliest range (or the ranges do not overlap).

This can be checked with a simple condition: MBRContains(rg1, StartPoint(rg2)).

Since there is no sargable condition that would find out which range starts earlier, we’ll just need to check both ways.

Unfortunately, MySQL does not support merging spatial indexes, so an OR condition would make the indexes unusable. We need to use the two MBRContains predicates with two separate queries and merge the results using UNION.

When optimizing a join, MySQL’s optimizer usually makes the smaller table leading in the join. With a usual equijoin condition this is a wise thing to do, since MySQL uses nested loops with an index search to join the tables. The duration of the nested loops query is the product of the leading table scan time (the number of the records) and driven table search time. B-Tree search is logarithmic, so (n × log m) is less than (m × log n) as long as m < n. That’s why the leading table should be the smallest one.

However, MySQL does not take into account that MBRContains predicate is not symmetrical. Reversing the table scan order makes each search operation to be not a searching for boxes containing the point, but searching for start points contained by the box. Since the individual points are not even indexed, MySQL will not be able to use an index access path for such a query and will revert to a very inefficient join buffer (which in fact is comparing every record to every other record).

To work around this, we need to force the join order so that the table with the points leads.

In MySQL, this is done by replacing JOIN with STRAIGHT_JOIN, which forces the leftmost table to be leading in the join.

Here’s the query:

SELECT  COUNT(*)
FROM    (
        SELECT  s.id AS sid, b.id AS bid
        FROM    t_small s
        STRAIGHT_JOIN
                t_big b
        ON      MBRContains(b.rg, StartPoint(s.rg))
        UNION
        SELECT  s.id AS sid, b.id AS bid
        FROM    t_big b
        STRAIGHT_JOIN
                t_small s
        ON      MBRContains(s.rg, StartPoint(b.rg))
        ) q
COUNT(*)
124911
1 row fetched in 0.0001s (4.4999s)
id select_type table type possible_keys key key_len ref rows filtered Extra
1 PRIMARY Select tables optimized away
2 DERIVED s ALL 5000 100.00
2 DERIVED b range sx_big_rg sx_big_rg 34 1 12000000.00 Range checked for each record (index map: 0×2)
3 UNION b ALL 120000 100.00
3 UNION s range sx_small_rg sx_small_rg 34 1 500000.00 Range checked for each record (index map: 0×2)
UNION RESULT <union2,3> ALL 
select count(0) AS `COUNT(*)` from (select `20100201_ranges`.`s`.`id` AS `sid`,`20100201_ranges`.`b`.`id` AS `bid` from `20100201_ranges`.`t_small` `s` straight_join `20100201_ranges`.`t_big` `b` union select `20100201_ranges`.`s`.`id` AS `sid`,`20100201_ranges`.`b`.`id` AS `bid` from `20100201_ranges`.`t_big` `b` straight_join `20100201_ranges`.`t_small` `s`) `q`

This query finds the total number of pairs of overlapping ranges from both tables and runs for 4.5 seconds which is quite efficient.

MBRIntersects: a more efficient solution

Normally I would put the words hope that helps and a link to the question form here, which I use to close the articles with the answers.

But when testing the solution, John found out that MBRIntersects function (which just checks intersection of two boxes without splitting them into start and end points) is sargable too and the query above is quite overcomplicated.

This function is symmetric, that is MBRIntersects(a, b) ≡ MBRIntersects(b, a). However, MySQL is not aware of that, and an index is used only against the column in the first argument to MBRIntersects. This means we have to make t_small to lead in the join and put its column into the second argument of the function.

This way, the column from the driven table (t_big) will be searched for the value provided from the leading table using the index on the driven table, which is exactly what we need.

Here’s the new updated query:

SELECT  COUNT(*)
FROM    t_small s
STRAIGHT_JOIN
        t_big b
ON      MBRIntersects(b.rg, s.rg)
COUNT(*)
124911
1 row fetched in 0.0001s (1.2656s)
id select_type table type possible_keys key key_len ref rows filtered Extra
1 SIMPLE s ALL sx_small_rg 5000 100.00
1 SIMPLE b ALL sx_big_rg 120000 100.00 Range checked for each record (index map: 0×2) 
select count(0) AS `COUNT(*)` from `20100201_ranges`.`t_small` `s` straight_join `20100201_ranges`.`t_big` `b` where intersects(`20100201_ranges`.`b`.`rg`,`20100201_ranges`.`s`.`rg`)

As we can see, this query is much more simple and completes 3 times as fast.

Finally, let’s see how can we retrieve the datetime values of the start and end of the ranges from the LineString’s:

SELECT  s.id AS small_id,
        '1970-01-01' + INTERVAL Y(StartPoint(s.rg)) SECOND AS small_start,
        '1970-01-01' + INTERVAL Y(EndPoint(s.rg)) SECOND AS small_end,
        b.id AS big_id,
        '1970-01-01' + INTERVAL Y(StartPoint(b.rg)) SECOND AS big_start,
        '1970-01-01' + INTERVAL Y(EndPoint(b.rg)) SECOND AS big_end
FROM    t_small s
STRAIGHT_JOIN
        t_big b
ON      MBRIntersects(b.rg, s.rg)
LIMIT 10
small_id small_start small_end big_id big_start big_end
1 2009-01-31 00:00:00 2009-02-01 20:58:03 1 2009-01-31 23:00:00 2009-02-01 00:52:25
1 2009-01-31 00:00:00 2009-02-01 20:58:03 2 2009-01-31 22:00:00 2009-01-31 22:01:55
1 2009-01-31 00:00:00 2009-02-01 20:58:03 3 2009-01-31 21:00:00 2009-01-31 21:32:21
1 2009-01-31 00:00:00 2009-02-01 20:58:03 4 2009-01-31 20:00:00 2009-01-31 20:36:01
1 2009-01-31 00:00:00 2009-02-01 20:58:03 5 2009-01-31 19:00:00 2009-01-31 20:23:00
1 2009-01-31 00:00:00 2009-02-01 20:58:03 6 2009-01-31 18:00:00 2009-01-31 19:06:56
1 2009-01-31 00:00:00 2009-02-01 20:58:03 7 2009-01-31 17:00:00 2009-01-31 18:25:41
1 2009-01-31 00:00:00 2009-02-01 20:58:03 8 2009-01-31 16:00:00 2009-01-31 17:47:38
1 2009-01-31 00:00:00 2009-02-01 20:58:03 9 2009-01-31 15:00:00 2009-01-31 15:41:06
1 2009-01-31 00:00:00 2009-02-01 20:58:03 10 2009-01-31 14:00:00 2009-01-31 14:02:38
10 rows fetched in 0.0006s (0.0024s)

This retrieves the range bounds in plain DATETIME format, suitable for output and calculations.

Hope that helps, John, and thanks for a nice tip!

I’m always glad to answer the questions regarding database queries.

Ask me a question

]]> http://explainextended.com/2010/02/01/join-on-overlapping-date-ranges/feed/ 0 Aggregates: subqueries vs. GROUP BY http://explainextended.com/2010/01/30/aggregates-subqueries-vs-group-by/ http://explainextended.com/2010/01/30/aggregates-subqueries-vs-group-by/#comments Sat, 30 Jan 2010 20:00:14 +0000 Quassnoi http://explainextended.com/?p=4044 From Stack Overflow:

I have a table users and there is a field invited_by_id showing user id of the person who invited this user.

I need to make a MySQL query returning rows with all the fields from users plus a invites_count field showing how many people were invited by each user.

The task seems very simple (and it is in fact), but there are at least two approaches to do it. It this article, I will discuss the benefits and the drawbacks of each approach.

GROUP BY

The first approach is using GROUP BY:

SELECT  u.*, COUNT(ui.id)
FROM    users u
LEFT JOIN
        users ui
ON      ui.invited_by = u.id
GROUP BY
        u.id

This is a GROUP BY with a self join, very simple. There are only two little things I’d like to pay some attention to.

First, we need to select all users, even those who invited no other members. An inner join would leave them out, so we use a LEFT JOIN, and use COUNT(ui.id) instead of COUNT(*), because, due to the very nature of aggregation, COUNT(*) returns at least 1 in a query with GROUP BY, and COUNT(ui.id) skips NULLs (which can only result from a LEFT JOIN miss).

Second, we group by u.id but use u.* in the SELECT clause. Every other engine would fail in this case, but MySQL allows selecting fields that are neighter grouped by nor aggregated. These fields will return an arbitrary value from any of the aggregated records (in practice, that is the record first read in its group).

This behavior is often abused, since MySQL does not guarantee what exactly it will return, but it’s perfectly valid for the queries like ours. We don’t need to clutter the GROUP BY clause with all fields from users if we have already grouped by the PRIMARY KEY which is already unique. All other values from users are uniquely defined by the PRIMARY KEY so there is no matter which arbitrary record will the query use to return ungrouped values: they are all same within the group.

Subquery

This solution involves correlated subqueries:

SELECT  u.*,
        (
        SELECT  COUNT(*)
        FROM    users ui
        WHERE   ui.invited_by = u.id
        )
FROM    users u

Here, we calculate the COUNT(*) in a correlated subquery. This query returns exactly same records as the previous one.

Comparison

Let’s create two sample tables (a MyISAM one and an InnoDB one) and see which solution is more efficient for different scenarios:

Table creation details

There are two identical tables, one using MyISAM storage engine, the other one using InnoDB. Each table contains 100,000 records.

The field invited_by is filled with a random value from 0 (including) to current id (not including). 0 means that the user was not invited (and really, who could invite the first user?).

All records

We well calculate the number of members invited by each user. For the sake of brevity we will aggregate the returned records.

MyISAM

GROUP BY

Subquery

We see that subquery solution is much more efficient: 3 seconds against 14 seconds, almost 5 times.

What is the reason?

If we look into the query plans we will see that MySQL uses temporary and filesort for the GROUP BY query.

MySQL knows only one way to aggregate the records, namely, sorting. To execute any GROUP BY statement it should first order the records according to GROUP BY conditions. This naturally returns records sorted by GROUP BY expressions and MySQL even cared to document this behavior.

There is a PRIMARY KEY index on id field, but traversing the indexes is quite slow in MySQL, so the optimizer preferred to do a filesort.

The subqueries, on the other hand, do not require any additional sorting, so retrieving the values aggregated in the subqueries is much faster. The records, of course, will not return in any specific order, but it wasn’t required anyway.

InnoDB

GROUP BY

Subquery

With InnoDB, both queries complete in almost same time, but the GROUP BY query is still a little bit faster.

Unlike MyISAM, with InnoDB tables the optimizer chooses the index access path which avoids GROUP BY sorting.

This is because InnoDB tables are index-organized, and the PRIMARY KEY is the table itself. So the table scan and the PRIMARY KEY scan are in fact the same thing in InnoDB and there is no point in additional sorting.

The algorithms behind the LEFT JOIN and the subqueries are in fact the same: just a single index range scan. However, due to some implementation issues, the subquery access requires some additional overhead which makes the subquery to run about 15% longer than its LEFT JOIN counterpart.

LIMIT with index

Let’s try the same queries, but now just return the first 100 records.

MyISAM

GROUP BY

Subquery

In this case, optimizer makes the query to use the index even despite the fact that the table uses MyISAM, because LIMIT makes index traversal cheaper than sorting.

However, for some strange reason, the engine still sorts the final resultset (which returns already sorted from the GROUP BY), and the GROUP BY query is 3 times less efficient than the subqueries.

InnoDB

GROUP BY

Subquery

With InnoDB tables, optimizer makes no final filesort, so both solutions take almost the same time (albeit the subqueries are several percent less efficient again).

LIMIT without index

Now, let’s run the queries that limit the resultsets ordered by an unindexed expression. Usually, this means ordering by RAND() to show, say, 10 random users.

MyISAM

GROUP BY

Subquery

The subqueries are notably faster (2 seconds for the subqueries against 4 seconds for the GROUP BY).

This demonstrates an interesting flaw in MySQL optimizer algorithm.

We remember that MySQL used the table scan with a sort for a query without a LIMIT, and an index scan for a query using a LIMIT.

In this case, MySQL’s optimizer sees two clauses: GROUP BY and LIMIT, and there is an index avaiable for GROUP BY id.

Taken together, they should make the index access path more efficient, and as we already saw in the previous section, they do, since normally LIMIT just takes first 10 records from the index.

But there is a third clause here, ORDER BY.

LIMIT of course gets applied to ORDER BY sorting, not GROUP BY one. MySQL’s optimizer, however, does not take this into account. Despite the fact that the ORDER BY will need the whole recordset, the index access path is still used. This makes GROUP BY to traverse all index. Of course no sorting is done (for GROUP BY that is), but traversal itself is quite slow in MyISAM.

The subquery solution, on the other hand, has only the ORDER BY and LIMIT. This makes the optimizer to choose the table scan which is much faster when we need all records.

InnoDB

GROUP BY

Subqueries

Formally, InnoDB is subject to the same optimizer mistake as the one described above for MyISAM. For the GROUP BY, we see the index access path in the query plan, for subqueries, we see ALL.

But for InnoDB, these two access paths are in fact the same, since an InnoDB table is a PRIMARY KEY and the index traversal over a PRIMARY KEY is a table traversal.

So unlike MyISAM, nothing bad happens in the first case here, and the GROUP BY query runs a little bit faster than the subqueries.

Summary

Generally speaking, aggregates over a left-joined table grouped by the main table’s primary key, and the aggregate subqueries over the same table yield the same results and use almost the same algorithms in MySQL.

However, they still differ performance-wise.

On the one hand, the subqueries require some overhead and are executed several percents more slowly than the joins.

On the other hand, GROUP BY in MySQL requires sorting the joined recordset on the GROUP BY expressions.

For InnoDB, the optimizer mostly makes the optimal decisions and does not sort the recordset, since an InnoDB table is always ordered by the PRIMARY KEY and the records naturally come in that order out of the table. The optimizer is aware of that.

For MyISAM, the PRIMARY KEY index does not contain the table values. The table needs to be looked up which requires extra work. That’s why in MyISAM the optimizer often makes incorrect decisions about whether or not use the index sort order or sort records taken from the table. These incorrect decisions lead to the subqueries being more efficient.

For MyISAM tables, the subqueries are often a better alternative to the GROUP BY.

For InnoDB tables, the subqueries and the GROUP BY complete in almost same time, but GROUP BY is still several percent more efficient.

With InnoDB, the GROUP BY queries over a left-joined table should be preferred over running the aggregate subqueries.

]]> http://explainextended.com/2010/01/30/aggregates-subqueries-vs-group-by/feed/ 0 Latest visitors http://explainextended.com/2010/01/12/latest-visitors/ http://explainextended.com/2010/01/12/latest-visitors/#comments Tue, 12 Jan 2010 20:00:26 +0000 Quassnoi http://explainextended.com/?p=3972 From Stack Overflow:

Say you want to display the latest visitors on a users profile page.

How would you structure this?

Perhaps a table called uservisitors:

uservisitors
userid visitorid time

And how would you select this with MySQL without any duplicates?

What I mean is if user 1 visits user 2’s profile, then 5 minutes later visits it again, I don’t want to show both entries: only the latest.

There are two approaches to this.

First one would be just aggregating the visits, finding the max time and ordering on it. Something like this:

SELECT  visitorid, MAX(time) AS lastvisit
FROM    uservisitors
WHERE   userid = 1
GROUP BY
        userid, visitorid
ORDER BY
        lastvisit DESC
LIMIT 5

However, there is a little problem with this solution.

Despite the fact that MySQL (with proper indexing) uses INDEX FOR GROUP-BY optimization for this query, it will still have to sort on MAX(time) to find 5 latest records.

This will require sorting the whole resultset which will be huge if the service is heavily loaded.

Let’s test it on a sample table:

Table creation details

This table has 1,000,000 records for 2 different users with 100,000 unique visitors. This gives 5 visits to each user’s page per visitor.

This reflects real distribution of the data on an actual site quite well.

The query we initially came up with gives the following results:

SELECT  visitorid, MAX(time) AS lastvisit
FROM    uservisitors
WHERE   userid = 2
GROUP BY
        userid, visitorid
ORDER BY
        lastvisit DESC
LIMIT 5
visitorid lastvisit
2 2009-01-11 23:58:00
4 2009-01-11 23:56:00
6 2009-01-11 23:54:00
8 2009-01-11 23:52:00
10 2009-01-11 23:50:00
5 rows fetched in 0.0002s (1.5469s)
id select_type table type possible_keys key key_len ref rows filtered Extra
1 SIMPLE uservisitors range ix_uservisitors_userid_visitorid_time,ix_uservisitors_userid_time ix_uservisitors_userid_visitorid_time 8 125068 100.00 Using where; Using index for group-by; Using temporary; Using filesort 
select `20100112_visitors`.`uservisitors`.`visitorid` AS `visitorid`,max(`20100112_visitors`.`uservisitors`.`time`) AS `lastvisit` from `20100112_visitors`.`uservisitors` where (`20100112_visitors`.`uservisitors`.`userid` = 2) group by `20100112_visitors`.`uservisitors`.`userid`,`20100112_visitors`.`uservisitors`.`visitorid` order by max(`20100112_visitors`.`uservisitors`.`time`) desc limit 5

This query uses INDEX FOR GROUP-BY, but it needs to make the sorting twice: first to calculate the aggregates, second to satisfy the ORDER BY clause.

An index scan can be used instead of the first sorting but the second sorting still needs to be done.

That’s why there is a filesort in the plan.

The query works for 1.5 seconds which is way too much for a busy service.

But since we only need the latest 5 visits to each user, we can just iterate the records using an index on time and filter out the duplicates until we return 5 records.

This can be done by appending a simple condition into the WHERE clause that would look back in time and find if there already were records with the same values of (userid, visitorid).

This is of course a variation of Schlemiel the painter’s algorithm.

However, since we only need 5 records and the duplicate visitors are very improbable, this should work alright.

Let’s check it:

SELECT  visitorid
FROM    uservisitors uo
WHERE   userid = 1
        AND NOT EXISTS
        (
        SELECT  NULL
        FROM    uservisitors ui
        WHERE   ui.userid = uo.userid
                AND (ui.time, ui.id) > (uo.time, uo.id)
                AND ui.visitorid = uo.visitorid
        )
ORDER BY
        userid DESC, time DESC
LIMIT 5
visitorid
1
3
5
7
9
5 rows fetched in 0.0001s (0.0018s)
id select_type table type possible_keys key key_len ref rows filtered Extra
1 PRIMARY uo ref ix_uservisitors_userid_visitorid_time,ix_uservisitors_userid_time ix_uservisitors_userid_time 4 const 22940 100.00 Using where
2 DEPENDENT SUBQUERY ui ref ix_uservisitors_userid_visitorid_time,ix_uservisitors_userid_time ix_uservisitors_userid_visitorid_time 8 20100112_visitors.uo.userid,20100112_visitors.uo.visitorid 4 100.00 Using where; Using index 
Field or reference '20100112_visitors.uo.userid' of SELECT #2 was resolved in SELECT #1
Field or reference '20100112_visitors.uo.time' of SELECT #2 was resolved in SELECT #1
Field or reference '20100112_visitors.uo.id' of SELECT #2 was resolved in SELECT #1
Field or reference '20100112_visitors.uo.visitorid' of SELECT #2 was resolved in SELECT #1
select `20100112_visitors`.`uo`.`visitorid` AS `visitorid` from `20100112_visitors`.`uservisitors` `uo` where ((`20100112_visitors`.`uo`.`userid` = 1) and (not(exists(select NULL AS `NULL` from `20100112_visitors`.`uservisitors` `ui` where ((`20100112_visitors`.`ui`.`userid` = `20100112_visitors`.`uo`.`userid`) and ((`20100112_visitors`.`ui`.`time`,`20100112_visitors`.`ui`.`id`) > (`20100112_visitors`.`uo`.`time`,`20100112_visitors`.`uo`.`id`)) and (`20100112_visitors`.`ui`.`visitorid` = `20100112_visitors`.`uo`.`visitorid`)))))) order by `20100112_visitors`.`uo`.`userid` desc,`20100112_visitors`.`uo`.`time` desc limit 5

Here we use NOT EXISTS clause to check whether the current visitor had a previous visit (and therefore was already returned by the query). This eliminates duplicates.

MySQL is not very good in optimizing the index access for correlated subqueries, that’s why for each visitorid it checks all records for the given (userid, visitorid) (despite the fact that we have an additional range condition on time which could be employed too). That’s why we see a USING WHERE in the plan for the DEPENDENT SUBQUERY.

However, since every visitor has only 5 visits to any given userid, this is not much of an issue.

The query completes in 1 ms which is next to instant.

Update of Jan 13th, 2010

As johno correctly pointed out, it is possible to use LEFT JOIN / IS NULL or NOT IN instead of NOT EXISTS to get the same results:

NOT IN

SELECT  visitorid
FROM    uservisitors uo
WHERE   userid = 1
        AND (userid, visitorid) NOT IN
        (
        SELECT  userid, visitorid
        FROM    uservisitors ui
        WHERE   (ui.time, ui.id) > (uo.time, uo.id)
        )
ORDER BY
        userid DESC, time DESC
LIMIT 5
visitorid
1
3
5
7
9
5 rows fetched in 0.0001s (0.0017s)
id select_type table type possible_keys key key_len ref rows filtered Extra
1 PRIMARY uo ref ix_uservisitors_userid_visitorid_time,ix_uservisitors_userid_time ix_uservisitors_userid_time 4 const 22940 100.00 Using where
2 DEPENDENT SUBQUERY ui ref ix_uservisitors_userid_visitorid_time,ix_uservisitors_userid_time ix_uservisitors_userid_visitorid_time 8 func,func 4 100.00 Using where; Using index 
Field or reference '20100112_visitors.uo.time' of SELECT #2 was resolved in SELECT #1
Field or reference '20100112_visitors.uo.id' of SELECT #2 was resolved in SELECT #1
select `20100112_visitors`.`uo`.`visitorid` AS `visitorid` from `20100112_visitors`.`uservisitors` `uo` where ((`20100112_visitors`.`uo`.`userid` = 1) and (not(<in_optimizer>((`20100112_visitors`.`uo`.`userid`,`20100112_visitors`.`uo`.`visitorid`),<exists>(select `20100112_visitors`.`ui`.`userid` AS `userid`,`20100112_visitors`.`ui`.`visitorid` AS `visitorid` from `20100112_visitors`.`uservisitors` `ui` where (((`20100112_visitors`.`ui`.`time`,`20100112_visitors`.`ui`.`id`) > (`20100112_visitors`.`uo`.`time`,`20100112_visitors`.`uo`.`id`)) and (<cache>(`20100112_visitors`.`uo`.`userid`) = `20100112_visitors`.`ui`.`userid`) and (<cache>(`20100112_visitors`.`uo`.`visitorid`) = `20100112_visitors`.`ui`.`visitorid`)) having (<is_not_null_test>(`20100112_visitors`.`ui`.`userid`) and <is_not_null_test>(`20100112_visitors`.`ui`.`visitorid`))))))) order by `20100112_visitors`.`uo`.`userid` desc,`20100112_visitors`.`uo`.`time` desc limit 5

LEFT JOIN / IS NULL

SELECT  uo.visitorid
FROM    uservisitors uo
LEFT JOIN
        uservisitors ui
ON      ui.userid = uo.userid
        AND (ui.time, ui.id) > (uo.time, uo.id)
        AND ui.visitorid = uo.visitorid
WHERE   uo.userid = 1
        AND ui.userid IS NULL
ORDER BY
        uo.userid DESC, uo.time DESC
LIMIT 5
visitorid
1
3
5
7
9
5 rows fetched in 0.0001s (0.0018s)
id select_type table type possible_keys key key_len ref rows filtered Extra
1 SIMPLE uo ref ix_uservisitors_userid_visitorid_time,ix_uservisitors_userid_time ix_uservisitors_userid_time 4 const 22940 100.00 Using where
1 SIMPLE ui ref ix_uservisitors_userid_visitorid_time,ix_uservisitors_userid_time ix_uservisitors_userid_visitorid_time 8 const,20100112_visitors.uo.visitorid 4 100.00 Using where; Using index; Not exists 
select `20100112_visitors`.`uo`.`visitorid` AS `visitorid` from `20100112_visitors`.`uservisitors` `uo` left join `20100112_visitors`.`uservisitors` `ui` on(((`20100112_visitors`.`ui`.`visitorid` = `20100112_visitors`.`uo`.`visitorid`) and (`20100112_visitors`.`ui`.`userid` = 1) and ((`20100112_visitors`.`ui`.`time`,`20100112_visitors`.`ui`.`id`) > (`20100112_visitors`.`uo`.`time`,`20100112_visitors`.`uo`.`id`)))) where ((`20100112_visitors`.`uo`.`userid` = 1) and isnull(`20100112_visitors`.`ui`.`userid`)) order by `20100112_visitors`.`uo`.`userid` desc,`20100112_visitors`.`uo`.`time` desc limit 5

As I mentioned in one of the previous articles:

, MySQL optimizes all these three predicates to use the same algorithm (return a FALSE of the first match), but the tests show that LEFT JOIN / IS NULL and NOT IN implement this algorithm a little bit more efficiently than NOT EXISTS.

Though on such a small resultset the difference is negligible, it is always better to use more efficient solutions.

]]> http://explainextended.com/2010/01/12/latest-visitors/feed/ 0 Linked lists in MySQL: multiple ordering http://explainextended.com/2009/12/29/linked-lists-in-mysql-multiple-ordering/ http://explainextended.com/2009/12/29/linked-lists-in-mysql-multiple-ordering/#comments Tue, 29 Dec 2009 20:00:37 +0000 Quassnoi http://explainextended.com/?p=3894 Answering questions asked on the site.

Rick McIntosh asks:

I have two tables, one with a list of spreadsheets (a) and the other with a list of the column headings that show in the spreadsheets (b).

a
id parent
1 2
2 0
b
id parent aid
1 1 1
2 0 1
3 2 1
4 6 2
5 4 2
6 0 2

I want to bring the columns back in first the order of the spreadsheets as defined by the a.parent_id then ordered as b.parent_id:

id parent aid aparent
6 0 2 0
4 6 2 0
5 4 2 0
2 0 1 2
3 2 1 2
1 1 1 2

This can be done using the same recursion technique as the one that was used to build a simple linked list.

MySQL does not support recursion directly, but it can be emulated using subquery calls in the SELECT clause of the query, using session variables to store the recursion state.

In this case we need to do the following:

  1. Generate a dummy recordset for recursion that would contain as many rows as the resulting recordset. This is best done by issuing a JOIN on the FOREIGN KEY, without any ordering. The values of the recordset will not be used in the actual query.

  2. Initialize @a and @b to be the first value of a and a 0, accordingly.

  3. In the loop, make a query that would return the next item of b for the current value of @a, or, should it fail, return the first item of b for the next @a. This is best done using a UNION ALL with a LIMIT.

  4. Adjust @a so that is points to the correct item: just select the appropriate value from b

Let’s create a sample table:

Table creation details

Here’s what this table contains (in id order):

SELECT  *
FROM    a
JOIN    b
ON      b.aid = a.id
ORDER BY
        a.id, b.id
id parent id parent aid
1 0 1 2 1
1 0 2 3 1
1 0 3 0 1
2 4 4 5 2
2 4 5 0 2
2 4 6 4 2
3 1 7 8 3
3 1 8 9 3
3 1 9 0 3
4 3 10 12 4
4 3 11 0 4
4 3 12 11 4
5 2 13 0 5
5 2 14 15 5
5 2 15 13 5
15 rows fetched in 0.0006s (0.0029s)

There are 5 records in a and 15 records in b (3 records per item).

Here’s the query:

SELECT  b.id, b.parent, a.id AS aid, a.parent AS aparent
FROM    (
        SELECT  @a AS _a,
                @b AS _b,
                @b :=
                (
                SELECT  id
                FROM    b
                WHERE   b.aid = _a
                        AND b.parent = _b
                UNION ALL
                SELECT  b.id
                FROM    a
                JOIN    b
                ON      b.aid = a.id
                WHERE   b.parent = 0
                        AND a.parent = _a
                LIMIT 1
                ) AS bid,
                @b AS _nb,
                @a :=
                (
                SELECT  aid
                FROM    b
                WHERE   b.id = _nb
                ) AS aid
        FROM    (
                SELECT  @a := a.id, @b := 0
                FROM    a
                WHERE   parent = 0
                ) vars,
                a
        JOIN    b
        ON      b.aid = a.id
) q
JOIN    a
ON      a.id = aid
JOIN    b
ON      b.id = bid
id parent aid aparent
3 0 1 0
2 3 1 0
1 2 1 0
9 0 3 1
8 9 3 1
7 8 3 1
11 0 4 3
12 11 4 3
10 12 4 3
5 0 2 4
4 5 2 4
6 4 2 4
13 0 5 2
15 13 5 2
14 15 5 2
15 rows fetched in 0.0006s (0.0073s)
id select_type table type possible_keys key key_len ref rows filtered Extra
1 PRIMARY a index PRIMARY ux_a_parent 4 5 100.00 Using index
1 PRIMARY <derived2> ALL 15 100.00 Using where; Using join buffer
1 PRIMARY b eq_ref PRIMARY PRIMARY 4 q.bid 1 100.00 Using where
2 DERIVED <derived6> system 1 100.00
2 DERIVED a index PRIMARY PRIMARY 4 5 100.00 Using index
2 DERIVED b ref ux_b_aid_parent ux_b_aid_parent 4 20091229_linked.a.id 7 100.00 Using index
6 DERIVED a const ux_a_parent ux_a_parent 4 1 100.00 Using index
5 DEPENDENT SUBQUERY b eq_ref PRIMARY PRIMARY 4 func 1 100.00 Using where
3 DEPENDENT SUBQUERY b eq_ref ux_b_aid_parent ux_b_aid_parent 8 func,func 1 100.00 Using where; Using index
4 DEPENDENT UNION a eq_ref PRIMARY,ux_a_parent ux_a_parent 4 func 1 100.00 Using where; Using index
4 DEPENDENT UNION b eq_ref ux_b_aid_parent ux_b_aid_parent 8 20091229_linked.a.id 1 100.00 Using where; Using index
UNION RESULT <union3,4> ALL 
Field or reference '_a' of SELECT #3 was resolved in SELECT #2
Field or reference '_b' of SELECT #3 was resolved in SELECT #2
Field or reference '_a' of SELECT #4 was resolved in SELECT #2
Field or reference '_nb' of SELECT #5 was resolved in SELECT #2
select `20091229_linked`.`b`.`id` AS `id`,`20091229_linked`.`b`.`parent` AS `parent`,`20091229_linked`.`a`.`id` AS `aid`,`20091229_linked`.`a`.`parent` AS `aparent` from (select (@a) AS `_a`,(@b) AS `_b`,(@b:=(select `20091229_linked`.`b`.`id` AS `id` from `20091229_linked`.`b` where ((`20091229_linked`.`b`.`aid` = `_a`) and (`20091229_linked`.`b`.`parent` = `_b`)) union all select `20091229_linked`.`b`.`id` AS `id` from `20091229_linked`.`a` join `20091229_linked`.`b` where () limit 1)) AS `bid`,(@b) AS `_nb`,(@a:=(select `20091229_linked`.`b`.`aid` AS `aid` from `20091229_linked`.`b` where (`20091229_linked`.`b`.`id` = `_nb`))) AS `aid` from (select (@a:='1') AS `@a := a.id`,(@b:=0) AS `@b := 0` from `20091229_linked`.`a` where 1) `vars` join `20091229_linked`.`a` join `20091229_linked`.`b` where (`20091229_linked`.`b`.`aid` = `20091229_linked`.`a`.`id`)) `q` join `20091229_linked`.`a` join `20091229_linked`.`b` where ((`20091229_linked`.`b`.`id` = `q`.`bid`) and (`20091229_linked`.`a`.`id` = `q`.`aid`))

This returns all rows in correct order.

Note that I used aliases (_a, _b and _nb) instead of variables in the subqueries. This helps MySQL to pick correct execution plans and use the indexes: a predicate containing a variable is an UNCACHEABLE SUBQUERY, while a predicate with an alias is a DEPENDENT SUBQUERY which is optimized much better.

Hope that helps.

I’m always glad to answer the questions regarding database queries.

Ask me a question

]]> http://explainextended.com/2009/12/29/linked-lists-in-mysql-multiple-ordering/feed/ 0 MySQL: Selecting records holding group-wise maximum (resolving ties) http://explainextended.com/2009/11/25/mysql-selecting-records-holding-group-wise-maximum-resolving-ties/ http://explainextended.com/2009/11/25/mysql-selecting-records-holding-group-wise-maximum-resolving-ties/#comments Wed, 25 Nov 2009 20:00:46 +0000 Quassnoi http://explainextended.com/?p=3759 Continuing the series on selecting records holding group-wise maximums.

The approach shown in the previous article is quite efficient, however, it can only be relied upon if the column being maximized is unique.

If the column is not unique (even along with the grouping column), the ties are possible: multiple records can satisfy the condition of holding a group-wise maximum.

In this article, I’ll show how to resolve the ties.

Let’s recreate the table we used in the previous article:

Table creation details

This table has 1,000,000 records:

  • id is the PRIMARY KEY
  • orderer is filled with random values from 1 to 10
  • glow is a low cardinality grouping field (10 distinct values)
  • ghigh is a high cardinality grouping field (10,000 distinct values)
  • stuffing is an asterisk-filled VARCHAR(200) column added to emulate payload of the actual tables

Now, let’s see how we deal with the different methods to resolve ties:

Returning all ties

Some tasks require all ties to be returned (that is, we need to return every record that holds the group-wise maximum or minimum).

This is simple and requires just a little modification to the query we used before. Instead of joining on the id, we just should join on the MAX or MIN value returned by the grouping query. This guarantees that every tie will be returned.

Here’s the query to return all group-wise maximums with ties for a high cardinality grouper:

SELECT  COUNT(*) AS cnt, SUM(id) AS idsum
FROM    (
        SELECT  ghigh, MAX(orderer) AS orderer
        FROM    t_distinct d
        GROUP BY
                ghigh
        ) dd
JOIN    t_distinct di
ON      di.ghigh = dd.ghigh
        AND di.orderer = dd.orderer

View query details

, and the same query for the low cardinality groupers:

SELECT  COUNT(*) AS cnt, SUM(id) AS idsum
FROM    (
        SELECT  glow, MAX(orderer) AS orderer
        FROM    t_distinct d
        GROUP BY
                glow
        ) dd
JOIN    t_distinct di
ON      di.glow = dd.glow
        AND di.orderer = dd.orderer

View query details

The final results are aggregated to avoid the post bloating, but the principle is the same.

We see that the queries return the same results but have different execution times.

Due to the way the data are distributed in the table, each grouper (both ghigh and glow) has at least one record with orderer = 1. The group-wise minimum for each row is, therefore, 1 and the queries above, in fact, just return all records with orderer = 1.

MySQL does all joins using the nested loops.

To return the values for high cardinality groupers (those that the table has many) it should do 10,000 loose index scans to find the group-wise minimums, each followed by 10,000 tight index scans returning all records with orderer = 1 (about 11 records per scan in average). This results in 111,431 records being returned.

The query against the low cardinality groupers does the same, but it performs 10 loose index scans to find the minimums and does 10 tight index scans (one per grouper) to find the records that hold it, returning 11,111 records per group.

Tight index scans are sequential and loose index scan require searching the B-Tree. Finding lots of sequential values in few groups is much faster than finding few sequential values in lots of groups, though both algorithms ultimately return the same records.

That’s why the query against the high cardinality groupers (lots of groups, few records per group) takes almost 450 ms, while the query against low cardinality groupers (few groups, lots of records per group) takes as few as 107 ms, or 4 times as fast.

Returning arbitrary record

In some occasions, we are not interested in the order of records within the tie: any single record would do as long as it holds the group-wise minimum or maximum.

This way we need to return any arbitrary record holding the group-wise maximum.

To do this, we can use the MySQL’s extension which allows to use the non-grouped and non-aggregates columns in a GROUP BY query.

When used this way, the expression will return an arbitrary value taken from any record within the group. This was designed to make the GROUP BY queries applied to joins, when both joining and grouping are done on the PRIMARY KEY of a table participating in the query. The join will duplicate the rows from the table and the GROUP BY will split them across the groups according to the value of the PRIMARY KEY. Since the PRIMARY KEY uniquely defines a row, all columns of the table are guaranteed to have the same values across the group, and there is no need to select any special one: any will do.

This could be applied to out task as well:

SELECT  COUNT(*), SUM(id), MAX(orderer), MIN(orderer)
FROM    (
        SELECT  di.*
        FROM    (
                SELECT  ghigh, MIN(orderer) AS orderer
                FROM    t_distinct d
                GROUP BY
                        ghigh
                ) dd
        JOIN    t_distinct di
        ON      di.ghigh = dd.ghigh
                AND di.orderer = dd.orderer
        GROUP BY
                di.ghigh, di.orderer
        ) q

View query details

This solution is quite efficient for the high-cardinality groupers, but has a serious drawback.

The expressions returned by the GROUP BY extension are not guaranteed to originate from one record.

Imagine this query:

SELECT  grouper, column1, column2
FROM    mytable
GROUP BY
         grouper

against this table:

grouper column1 column2
1 10 100
1 20 200

This query is free to return any value of column1 and column2 within the group.

That means that this resultset:

grouper column1 column2
1 10 200

would be a correct output for this query, with the values of column1 and column2 being taken from two different records (within a single group).

Though in reality MySQL returns the values from a single record, it’s a bad practice to rely on this behavior which can be changed with any new version or even a bugfix patch.

However, we can use a cleaner version of the query which would do just the same.

Instead of grouping on the non-primary key values, we can just select the primary key of an arbitrary record and join the table back on it.

To select a primary key of an arbitrary record we can just use a subquery with a LIMIT 1 applied to it.

Here’s the query for the high cardinality grouper:

SELECT  COUNT(*), SUM(id), MAX(orderer), MIN(orderer)
FROM    (
        SELECT  di.*
        FROM    (
                SELECT  ghigh, MIN(orderer) AS orderer
                FROM    t_distinct d
                GROUP BY
                        ghigh
                ) dd
        JOIN    t_distinct di
        ON      di.id =
                (
                SELECT  id
                FROM    t_distinct ds
                WHERE   ds.ghigh = dd.ghigh
                        AND ds.orderer = dd.orderer
                LIMIT 1
                )
        ) q

View query details

and the same query for the low cardinality grouper:

SELECT  di.id, di.orderer, di.ghigh, di.glow
FROM    (
        SELECT  glow, MIN(orderer) AS orderer
        FROM    t_distinct d
        GROUP BY
                glow
        ) dd
JOIN    t_distinct di
ON      di.id =
        (
        SELECT  id
        FROM    t_distinct ds
        WHERE   ds.glow = dd.glow
                AND ds.orderer = dd.orderer
        LIMIT 1
        )
id orderer ghigh glow
11 1 11 1
382 1 382 2
3 1 3 3
44 1 44 4
55 1 55 5
26 1 26 6
7 1 7 7
118 1 118 8
19 1 19 9
150 1 150 10
10 rows fetched in 0.0004s (0.0032s)

View query details

The low cardinality query returns much less columns and is almost instant (3 ms).

The high cardinality columns requires more index lookups that the previous GROUP BY version (to find the group-wise minimums; to find the id of an arbitrary record; to join the table back on this id). However, it does not use filesort.

This makes it just a little bit less efficient than a GROUP BY query (570 ms vs 560 ms), but it is guaranteed to return correct results, and, hence, should be used instead of the GROUP BY version.

Returning a record with maximum id

Finally, let’s make a query that would resolve the ties by returning the column with the maximum or the minimum id (or another unique column).

In this case we cannot get away with only using a MIN or a MAX, since we do need to sort on a composite condition, and MAX and MIN only accept and return a single value.

To do this, we need to use a subquery returning an id, just like we did in the previous case. The only thing we need to change is to add a correct ORDER BY condition to this subquery.

To make it more interesting, let’s create a query that would return the records holding the group-wise minimums of orderer, but would resolve ties by selecting a maximal id.

SELECT  di.id, di.orderer, di.ghigh, di.glow
FROM    (
        SELECT  glow, MIN(orderer) AS orderer
        FROM    t_distinct d
        GROUP BY
                glow
        ) dd
JOIN    t_distinct di
ON      di.id =
        (
        SELECT  id
        FROM    t_distinct ds
        WHERE   ds.glow = dd.glow
                AND ds.orderer = dd.orderer
        ORDER BY
                id DESC
        LIMIT 1
        )

View query details

This works, but is quite inefficient (380 ms).

This is due to the flaw in MySQL’s optimizer.

As you can see, the subquery has a filtering equality condition on two columns (glow and orderer) and an ordering condition on the third column, id.

There is an index that covers all three columns: ix_distinct_glow_orderer (glow, orderer). The first two columns are explicitly provided in the index definition, and the third column, id, being a PRIMARY KEY, is implicitly included by the InnoDB engine as a row pointer.

However, MySQL is able to use this index for ordering in a plain query but not in a subquery. It does use the index for filtering but orders using a filesort.

To work around this, we need to modify the ordering condition so that it would include all three columns constituting the index, all in the same direction:

SELECT  di.id, di.orderer, di.ghigh, di.glow
FROM    (
        SELECT  glow, MIN(orderer) AS orderer
        FROM    t_distinct d
        GROUP BY
                glow
        ) dd
JOIN    t_distinct di
ON      di.id =
        (
        SELECT  id
        FROM    t_distinct ds
        WHERE   ds.glow = dd.glow
                AND ds.orderer = dd.orderer
        ORDER BY
                glow DESC, orderer DESC, id DESC
        LIMIT 1
        )

View query details

Now there is no filesort in the DEPENDENT SUBQUERY, and the whole query completes in only 3 ms (100 times as fast).

The use of orderer DESC in the ordering condition of the subquery despite the fact that we need to return the group-wise minimum, not maximum may seem counterintuitive. However, this is only a workaround to make the ordering condition eligible for the index. The actual value of the orderer used the subquery is defined by the value of MAX or MIN returned by the outer query and the records are filtered on it anyway. Logically, orderer ASC, orderer DESC and even the absence of the orderer in the ORDER BY are the same. It’s just required to make MySQL use the index, and to do this we need to sort on all columns constituting the index, even though the first two ordering conditions are made constant by the filter and could be omitted as well.

To be continued.

]]> http://explainextended.com/2009/11/25/mysql-selecting-records-holding-group-wise-maximum-resolving-ties/feed/ 0 MySQL: Selecting records holding group-wise maximum (on a unique column) http://explainextended.com/2009/11/24/mysql-selecting-records-holding-group-wise-maximum-on-a-unique-column/ http://explainextended.com/2009/11/24/mysql-selecting-records-holding-group-wise-maximum-on-a-unique-column/#comments Tue, 24 Nov 2009 20:00:51 +0000 Quassnoi http://explainextended.com/?p=3736 Answering questions asked on the site.

Juan asks:

Regarding this question:

I would like to know what database is the most efficient when performing this type of query and if the solutions proposed are already the most efficient.

I am using MySQL but I would like to know if PostgreSQL, Oracle, Microsoft SQL Server or DB2 are much more efficient.

A nice question, this is a common problem for most developers.

For those too lazy to follow the link, the problem in one sentence:

How do I select the whole records, grouped on grouper and holding a group-wise maximum (or minimum) on other column?

I’ve already covered this in passing in various previous articles in my blog, and now I’ll try to make a little summary.

According to the SQL standards, every expression used in the SELECT list of a query that deals with GROUP BY should be either grouped by (that is, used in the GROUP BY) or aggregated.

The reason behind this is quite logical: a grouping query takes the records that have something in common and shrinks them, making a single record out of the whole set constituting the group.

The grouping expressions, of course, share their values across all the records constituting the group (since they are what that makes the group a group), and hence can be used as is.

Other expressions can vary and therefore some algorithm should be applied for shrinking all existing values into a single value. This is what aggregating function does.

The problem is that the the aggregating functions are not interdependent: you cannot use the result of one function as an input to another one. Hence, you cannot select a record in an aggregate query: you can only select a tuple of aggregated values. It’s possible that each of these aggregated values in fact can never even found among the values of the original records. Say, with all values of column being strictly positive, you can never find a record holding a SUM(column): it will be greater than any of the values of column.

However, with MIN or MAX, it is guaranteed that at least one record holds the non-NULL value returned by these aggregate functions. The problem is to return the whole record without aggregating the other columns.

As SQL evolved, different solutions had been proposed to solve this problem, and I’ll try to cover these solutions in this series of articles.

First, I’ll cover MySQL (as per request)

MySQL’s dialect of SQL is much more poor than those used by the other systems. That’s why most of the solutions used by the other systems, like analytic functions, DISTINCT ON etc. cannot be used in MySQL.

However, MySQL still provides some ways to make the queries that solve this task more efficient.

Let’s create a sample table:

Table creation details

This table has 1,000,000 records:

  • id is the PRIMARY KEY
  • orderer is filled with random values from 1 to 10
  • glow is a low cardinality grouping field (10 distinct values)
  • ghigh is a high cardinality grouping field (10,000 distinct values)

There are composite indexes on grouping columns: those including the id as a secondary field and those including the orderer.

We will split the task into four large categories which depend on properties the column we maximize (or minimize) on and how should we process ties.

The column being maximized is unique

This means that the ties are impossible: the value of the column is guaranteed to identify the row (even outside the group).

All we have to do is to select the greatest value of the column and join the table back on this value.

Here’s how we do it in MySQL:

SELECT  di.id, di.orderer, di.ghigh, di.glow
FROM    (
        SELECT  glow, MAX(id) AS id
        FROM    t_distinct d
        GROUP BY
                glow
        ) dd
JOIN    t_distinct di
ON      di.id = dd.id

View query details

In the query plan we see Using index for group-by.

This access method is used by MySQL to serve DISTINCT and GROUP BY queries, or, being more exact, to build the list of distinct groupers using an index.

The idea behind the method is very simple:

  1. Find the first key for the index and return it
  2. Iteratively find the next key (greater than the one previously found) traversing the index tree and return it

Whenever the next index key is found, the aggregates are applied to that key.

The indexes in MySQL are built using a B-Tree. This means that the iterative searching will have to restart from the root key on each key. The B-Tree is O(log(n)) levels deep in average, this means that the index scan for group by is O(log(n)) times as long as the index scan on these value would take be the values unique.

Since the aggregates normally require all values from the group, there is no point in using this access method to calculate the aggregates: a sequential index scan would be way more efficient (without O(log(n)) in the formula).

MIN and MAX, however, are somewhat special aggregates, since aggregating a column with them means just taking a first (or a last) value from the appropriate index. MySQL is aware of this and uses the index for group-by when calculating MIN and MAX on secondary index columns. This is called loose index scan.

The index for group-by, when used over a composite index, naturally stops on the lowest group-wise value of the second column. The MIN, therefore, is free: the value of the second column (which constitutes the MIN) should be just returned along with the value of the grouping column.

A MAX, on the other hand, requires an additional index search to find the last value of the group.

Overall query time, thus, depends on the grouping column cardinality and on the aggregate function (MIN or MAX being used). The lower the cardinality is, the better; and MIN is faster than MAX.

We see that on a low cardinality column, glow, which has only 10 distinct values, the query takes only 2 ms, which is next instant. We can hardly notice the difference between the MIN and MAX on such a fast query.

Let’s see the difference between two queries being grouped on a high cardinality column. Since the query would return 10,000 records, I’ll make it aggregate the final results as well to avoid bloating the post.

Here’s the MIN query:

SELECT  SUM(di.id)
FROM    (
        SELECT  ghigh, MIN(id) AS id
        FROM    t_distinct d
        GROUP BY
                ghigh
        ) dd
JOIN    t_distinct di
ON      di.id = dd.id
SUM(di.id)
50005000
1 row fetched in 0.0001s (0.1638s)
id select_type table type possible_keys key key_len ref rows filtered Extra
1 PRIMARY <derived2> ALL 10000 100.00
1 PRIMARY di eq_ref PRIMARY PRIMARY 4 dd.id 1 100.00 Using index
2 DERIVED d range ix_distinct_ghigh_id 4 20835 100.00 Using index for group-by 
select sum(`20091124_groupwise`.`di`.`id`) AS `SUM(di.id)` from (select `20091124_groupwise`.`d`.`ghigh` AS `ghigh`,min(`20091124_groupwise`.`d`.`id`) AS `id` from `20091124_groupwise`.`t_distinct` `d` group by `20091124_groupwise`.`d`.`ghigh`) `dd` join `20091124_groupwise`.`t_distinct` `di` where (`20091124_groupwise`.`di`.`id` = `dd`.`id`)

, and here’s the MAX one:

SELECT  SUM(di.id)
FROM    (
        SELECT  ghigh, MAX(id) AS id
        FROM    t_distinct d
        GROUP BY
                ghigh
        ) dd
JOIN    t_distinct di
ON      di.id = dd.id
SUM(di.id)
9950005000
1 row fetched in 0.0001s (0.2800s)
id select_type table type possible_keys key key_len ref rows filtered Extra
1 PRIMARY <derived2> ALL 10000 100.00
1 PRIMARY di eq_ref PRIMARY PRIMARY 4 dd.id 1 100.00 Using index
2 DERIVED d range ix_distinct_ghigh_id 4 20835 100.00 Using index for group-by 
select sum(`20091124_groupwise`.`di`.`id`) AS `SUM(di.id)` from (select `20091124_groupwise`.`d`.`ghigh` AS `ghigh`,max(`20091124_groupwise`.`d`.`id`) AS `id` from `20091124_groupwise`.`t_distinct` `d` group by `20091124_groupwise`.`d`.`ghigh`) `dd` join `20091124_groupwise`.`t_distinct` `di` where (`20091124_groupwise`.`di`.`id` = `dd`.`id`)

With the grouping column having high cardinality, this query needs much more next key lookups to complete and takes significantly more time. However, on a 1,000,000 records table, it’s still a matter of fractions of a second. The MAX query takes 280 ms, which is almost twice the time of the MIN query (160 ms).

As was just described, the reason for that is that the MAX query need an extra lookup, since the records being iterated by index for group-by hold the MIN, not the MAX.

Unfortunately, MySQL cannot iterate the index backwards in a loose index scan, nor it supports the DESC clause in the indexes. That means that taking the group-wise maximum is less efficient than taking a group-wise minimum (with a loose index scan being used).

However, with the grouping columns having sufficiently low cardinality, both queries are very fast and the difference negligible.

To be continued

]]> http://explainextended.com/2009/11/24/mysql-selecting-records-holding-group-wise-maximum-on-a-unique-column/feed/ 0