I want SQL to return blank row even if the condition does not match

This may be useful for certain ORMs which always expect a single row as a result of a query.

Say, we have a query like that:

SELECT  *
FROM    mytable
WHERE   id = 42

and want it to return a single row (possibly consisting of NULL values) no matter what.

If we had a join and the condition in the ON clause:

SELECT  m.*
FROM    values v
JOIN    mytable m
ON      m.id = v.value

, we could just rewrite an INNER JOIN to a LEFT JOIN.

SELECT  m.*
FROM    values v
LEFT JOIN
        mytable m
ON      m.id = v.value

This way, we would have at least one record returned for each entry in values.

In our original query we don’t have a table to join with. But we can easily generate it:

SELECT  m.*
FROM    (
        SELECT  42 AS value
        ) v
LEFT JOIN
        mytable m
ON      m.id = v.value

If id is a PRIMARY KEY on mytable, this query would return exactly one record, regardless of whether it such an id exists in mytable or not.

I have a query where I have a custom developed UDF that is used to calculate whether or not certain points are within a polygon (first query in UNION) or circular (second query in UNION) shape.

SELECT  a.geo_boundary_id, …
FROM     geo_boundary_vertex a, …
…
GROUP BY
        a.geo_boundary_id
UNION
SELECT  b.geo_boundary_id, …
FROM     geo_boundary b, …
…
GROUP BY
        b.geo_boundary_id

When I run an explain for the query I get filesort for both queries within the UNION.

Now, I can split the queries up and use the ORDER BY NULL trick to get rid of the filesort however when I attempt to add that to the end of a UNION it doesn’t work.

How do I get rid of the filesort?

In MySQL, GROUP BY also implies ORDER BY on the same set of expressions in the same order. That’s why it adds an additional filesort operation to sort the resultset if it does not come out naturally sorted (say, from an index).

This is not always a desired behavior, and MySQL manual suggests adding ORDER BY NULL to the queries where sorting is not required. This can improve performance of the queries.

Let’s create a sample table and see:

Table creation details

CREATE TABLE filler (
        id INT NOT NULL PRIMARY KEY AUTO_INCREMENT
) ENGINE=Memory;

CREATE TABLE grouping (
        id INT NOT NULL PRIMARY KEY AUTO_INCREMENT,
        value1 INT NOT NULL,
        value2 INT NOT NULL
) ENGINE=InnoDB;

DELIMITER $$

CREATE PROCEDURE prc_filler(cnt INT)
BEGIN
        DECLARE _cnt INT;
        SET _cnt = 1;
        WHILE _cnt <= cnt DO
                INSERT
                INTO    filler
                SELECT  _cnt;
                SET _cnt = _cnt + 1;
        END WHILE;
END
$$

DELIMITER ;

START TRANSACTION;
CALL prc_filler(100000);
COMMIT;

INSERT
INTO    grouping (value1, value2)
SELECT  CEILING(RAND(20110330) * 300000),
        CEILING(RAND(20110330 << 1) * 300000)
FROM    filler
CROSS JOIN
        (
        SELECT  id
        FROM    filler
        LIMIT 30
        ) q;

The table contains 3,000,000 random records with value1 and value2 between 1 and 300,000.

Here’s the plan we get with a mere UNION of two GROUP BY queries:

SELECT  value1 AS value
FROM    grouping
GROUP BY
        value1
UNION
SELECT  value2 AS value
FROM    grouping
GROUP BY
        value2
LIMIT 10

select `20110330_group`.`grouping`.`value1` AS `value` from `20110330_group`.`grouping` group by `20110330_group`.`grouping`.`value1` union select `20110330_group`.`grouping`.`value2` AS `value` from `20110330_group`.`grouping` group by `20110330_group`.`grouping`.`value2` limit 10

We see that there is a filesort in each of the queries.

MySQL does allow using ORDER BY in the queries merged with UNION or UNION ALL. To do this, we just need to wrap each query into a set of parentheses:

(
SELECT  value1 AS value
FROM    grouping
GROUP BY
        value1
ORDER BY
        NULL
)
UNION
(
SELECT  value2 AS value
FROM    grouping
GROUP BY
        value2
ORDER BY
        NULL
)
LIMIT 10

(select `20110330_group`.`grouping`.`value1` AS `value` from `20110330_group`.`grouping` group by `20110330_group`.`grouping`.`value1` order by NULL) union (select `20110330_group`.`grouping`.`value2` AS `value` from `20110330_group`.`grouping` group by `20110330_group`.`grouping`.`value2` order by NULL) limit 10

However, the plan remained the same. Why?

MySQL‘s documentation says:

Use of ORDER BY for individual SELECT statements implies nothing about the order in which the rows appear in the final result because UNION by default produces an unordered set of rows. Therefore, the use of ORDER BY in this context is typically in conjunction with LIMIT, so that it is used to determine the subset of the selected rows to retrieve for the SELECT, even though it does not necessarily affect the order of those rows in the final UNION result. If ORDER BY appears without LIMIT in a SELECT, it is optimized away because it will have no effect anyway.

This means that the optimizer just removes ORDER BY from the UNION parts if they are not used along with LIMIT.

This is of course a good idea: since individual ORDER BY have no effect on the order of the final query anyway, there is no use in executing them or even taking them into account.

However, this idea would be much better if the same were also true for GROUP BY. Currently, the optimizer does not optimize away the ordering behavior of the GROUP BY queries which are parts of a UNION and they cannot be cured with ORDER BY NULL (whose only goal is not to order) since this is removed by the optimizer.

However, since only ORDER BY clauses not accompanied with a LIMIT are thrown away, we could just add a LIMIT. Of course it should be large enough to guarantee that all record would be returned.

Let’s see:

(
SELECT  value1 AS value
FROM    grouping
GROUP BY
        value1
ORDER BY
        NULL
LIMIT 10000000000
)
UNION
(
SELECT  value2 AS value
FROM    grouping
GROUP BY
        value2
ORDER BY
        NULL
LIMIT 10000000000
)
LIMIT 10

(select `20110330_group`.`grouping`.`value1` AS `value` from `20110330_group`.`grouping` group by `20110330_group`.`grouping`.`value1` order by NULL limit 10000000000) union (select `20110330_group`.`grouping`.`value2` AS `value` from `20110330_group`.`grouping` group by `20110330_group`.`grouping`.`value2` order by NULL limit 10000000000) limit 10

Now, there are no filesort operations in the plan and the query runs 20% faster.

This is not a very elegant solution of course. More than that, a solution similar to it was used for SQL Server 2000 which does not allow using ORDER BY without a TOP in the inline views. In SQL Server 2000, TOP 100% forced the order of the nested queries and usually made them spooled or otherwise materialized.

It was quite a nasty surprise when SQL Server 2005 has improved its optimizer to detect such tricks and ignore ORDER BY for TOP 100% queries. However, with all improvements introduced in SQL Server 2005, most of these queries could be just rewritten in a more clean and efficient way.

Nevertheless, this solution is still safe to use, because it does not change the semantics of the query (if LIMIT is chosen large enough), but is just an optimizer hack. In the worst case, the query will just become as slow as it initially was, and an extra filesort is not the worst of all things that can happen to a query.

Meanwhile, I’ve posted it as bug 60702 to MySQL bug database and am hoping they’ll fix it in the next release.

Victor asks:

I have a table which I will call sale to protect the innocent:

Sale

id	product	price	amount	date

I need to retrieve ultimate values of price, amount and date for each product:

SELECT  product,
        MIN(price), MAX(price),
        MIN(amount), MAX(amount),
        MIN(date), MAX(date)
FROM    sale
GROUP BY
        product

The query only returns about 100 records.

I have all these fields indexed (together with product), but this still produces a full table scan over 3,000,000 records.

How do I speed up this query?

To retrieve the ultimate values of the fields, MySQL would just need to make a loose index scan over each index and take the max and min value of the field for each product.

However, the optimizer won’t do when multiple indexes are involved. Instead, it will revert to a full scan.

There is a workaround for it. Let’s create a sample table and see them:

Table creation details

CREATE TABLE filler (
        id INT NOT NULL PRIMARY KEY AUTO_INCREMENT
) ENGINE=Memory;

CREATE TABLE sale
        (
        id INT NOT NULL PRIMARY KEY AUTO_INCREMENT,
        product INT NOT NULL,
        amount INT NOT NULL,
        price NUMERIC(20, 2) NOT NULL,
        dt DATETIME NOT NULL,
        KEY ix_sale_product_amount (product, amount),
        KEY ix_sale_product_price (product, price),
        KEY ix_sale_product_dt (product, dt)
        )
ENGINE=InnoDB;

DELIMITER $$

CREATE PROCEDURE prc_filler(cnt INT)
BEGIN
        DECLARE _cnt INT;
        SET _cnt = 1;
        WHILE _cnt <= cnt DO
                INSERT
                INTO    filler
                SELECT  _cnt;
                SET _cnt = _cnt + 1;
        END WHILE;
END
$$

DELIMITER ;

START TRANSACTION;
CALL prc_filler(50000);
COMMIT;

INSERT
INTO    sale (product, amount, price, dt)
SELECT  CEILING(RAND(20110227) * 100),
        CEILING(RAND(20110227 << 1) * 1000) + 100,
        CEILING(RAND(20110227 << 1 + 1) * 10000) / 100.00 + 20.00,
        '2011-03-27' - INTERVAL CEILING(RAND(20110227 << 1 + 2) * 10000000) SECOND
FROM    filler
CROSS JOIN
        (
        SELECT  id
        FROM    filler
        ORDER BY
                id
        LIMIT 60
        ) q;

This table contains 3M random records for 100 distinct products and is indexed appropriately.

Here’s a straightforward query:

SELECT  product,
        MIN(price), MAX(price),
        MIN(amount), MAX(amount),
        MIN(dt), MAX(dt)
FROM    sale
GROUP BY
        product

select `20110327_split`.`sale`.`product` AS `product`,min(`20110327_split`.`sale`.`price`) AS `MIN(price)`,max(`20110327_split`.`sale`.`price`) AS `MAX(price)`,min(`20110327_split`.`sale`.`amount`) AS `MIN(amount)`,max(`20110327_split`.`sale`.`amount`) AS `MAX(amount)`,min(`20110327_split`.`sale`.`dt`) AS `MIN(dt)`,max(`20110327_split`.`sale`.`dt`) AS `MAX(dt)` from `20110327_split`.`sale` group by `20110327_split`.`sale`.`product`

As was expected, the query uses a full table scan (or, rather, an index scan) and takes almost 14 seconds to complete.

In order to make the engine use three separate indexes, we would need to split the query into three queries each searching for ultimate values on one column, and then combine the results.

Each of these queries would be a MIN / MAX query on a trailing column of a composite index, combined with a GROUP BY on the index’s leading columns, and, as such, would be subject to loose index scan optimization.

To combine the results, we will of course just join them on product. This will be quite efficient too since the resultsets are going to be quite small (100 records each, exactly).

And here is the query:

SELECT  *
FROM    (
        SELECT  product, MIN(amount), MAX(amount)
        FROM    sale
        GROUP BY
                product
        ) qa
NATURAL JOIN
        (
        SELECT  product, MIN(price), MAX(price)
        FROM    sale
        GROUP BY
                product
        ) qp
NATURAL JOIN
        (
        SELECT  product, MIN(dt), MAX(dt)
        FROM    sale
        GROUP BY
                product
        ) qd

select `qa`.`product` AS `product`,`qa`.`MIN(amount)` AS `MIN(amount)`,`qa`.`MAX(amount)` AS `MAX(amount)`,`qp`.`MIN(price)` AS `MIN(price)`,`qp`.`MAX(price)` AS `MAX(price)`,`qd`.`MIN(dt)` AS `MIN(dt)`,`qd`.`MAX(dt)` AS `MAX(dt)` from (select `20110327_split`.`sale`.`product` AS `product`,min(`20110327_split`.`sale`.`amount`) AS `MIN(amount)`,max(`20110327_split`.`sale`.`amount`) AS `MAX(amount)` from `20110327_split`.`sale` group by `20110327_split`.`sale`.`product`) `qa` join (select `20110327_split`.`sale`.`product` AS `product`,min(`20110327_split`.`sale`.`price`) AS `MIN(price)`,max(`20110327_split`.`sale`.`price`) AS `MAX(price)` from `20110327_split`.`sale` group by `20110327_split`.`sale`.`product`) `qp` join (select `20110327_split`.`sale`.`product` AS `product`,min(`20110327_split`.`sale`.`dt`) AS `MIN(dt)`,max(`20110327_split`.`sale`.`dt`) AS `MAX(dt)` from `20110327_split`.`sale` group by `20110327_split`.`sale`.`product`) `qd` where ((`qp`.`product` = `qa`.`product`) and (`qd`.`product` = `qa`.`product`))

Each individual query is executed with using index for group-by, that is jumping over the min and max values in a loose index scan.

The queries are combined in a join which, despite not using the indexes, is still very fast because only 100 records are joined (of course fitting into a join buffer).

The overall query time is only 10 ms.

Hope that helps.

I’m always glad to answer the questions regarding database queries.

Ask me a question

Aryé asks:

Thanks for your article about late row lookups in MySQL.

I have two questions for you please:

• Is this workaround specific to MyISAM engine?
• How does PostgreSQL handle this?

The questions concerns a certain workaround for MySQL LIMIT … OFFSET queries like this:

SELECT  *
FROM    mytable
ORDER BY
        id
LIMIT   10
OFFSET  10000

which can be improved using a little rewrite:

SELECT  m.*
FROM    (
        SELECT  id
        FROM    mytable
        ORDER BY
                id
        LIMIT   10
        OFFSET  10000
        ) q
JOIN    mytable m
ON      m.id = q.id
ORDER BY
        m.id

For the rationale behind this improvement, please read the original article.

Now, to the questions.

The second questions is easy: PostgreSQL won’t pull the fields from the table until it really needs them. If a query involving an ORDER BY along with LIMIT and OFFSET is optimized to use the index for the ORDER BY part, the table lookups won’t happen for the records skipped.

Though PostgreSQL does not reflect the table lookups in the EXPLAIN output, a simple test would show us that they are done only LIMIT times, not OFFSET + LIMIT (like MySQL does).

Now, let’s try to answer the first question: will this trick improve the queries against an InnoDB table?

To do this, we will create a sample table:

Table creation details

CREATE TABLE filler (
        id INT NOT NULL PRIMARY KEY AUTO_INCREMENT
) ENGINE=Memory;

CREATE TABLE lookup (
        id INT NOT NULL PRIMARY KEY,
        value INT NOT NULL,
        shorttxt TEXT NOT NULL,
        longtxt TEXT NOT NULL
) ENGINE=InnoDB ROW_FORMAT=COMPACT;

CREATE INDEX
        ix_lookup_value
ON      lookup (value);

DELIMITER $$

CREATE PROCEDURE prc_filler(cnt INT)
BEGIN
        DECLARE _cnt INT;
        SET _cnt = 1;
        WHILE _cnt <= cnt DO
                INSERT
                INTO    filler
                SELECT  _cnt;
                SET _cnt = _cnt + 1;
        END WHILE;
END
$$

DELIMITER ;

START TRANSACTION;
CALL prc_filler(100000);
COMMIT;

INSERT
INTO    lookup
SELECT  id, CEILING(RAND(20110211) * 1000000),
        RPAD('', CEILING(RAND(20110211 << 1) * 100), '*'),
        RPAD('', CEILING(8192 + RAND(20110211 << 1) * 100), '*')
FROM    filler;

There is one indexed INT column and two TEXT columns, shorttxt storing short strings (1 to 100 characters), longtxt storing long strings (8193 to 8293 characters).

Let’s run some queries against this table.

PRIMARY KEY and the INT column

No rewrite:

SELECT  value
FROM    lookup
ORDER BY
        id
LIMIT   10
OFFSET  90000

Query results

select `20110211_late`.`lookup`.`value` AS `value` from `20110211_late`.`lookup` order by `20110211_late`.`lookup`.`id` limit 90000,10

Rewrite:

Query results

SELECT  l.value
FROM    (
        SELECT  id
        FROM    lookup
        ORDER BY
                id
        LIMIT   10
        OFFSET  90000
        ) q
JOIN    lookup l
ON      l.id = q.id
ORDER BY
        q.id

select `20110211_late`.`l`.`value` AS `value` from (select `20110211_late`.`lookup`.`id` AS `id` from `20110211_late`.`lookup` order by `20110211_late`.`lookup`.`id` limit 90000,10) `q` join `20110211_late`.`lookup` `l` where (`20110211_late`.`l`.`id` = `q`.`id`) order by `q`.`id`

As you can see, there is almost no difference (41 ms vs 40 ms).

InnoDB tables are clustered on their PRIMARY KEY columns, which means the the index on id (used to serve the ORDER BY condition) contains all the data the query needs. There is a negligible benefit from not lookup up the value columns at the index pages because the column is tiny and the index pages need to be read anyway.

PRIMARY KEY and the short TEXT column

Rewrite:

SELECT  LENGTH(l.shorttxt)
FROM    (
        SELECT  id
        FROM    lookup
        ORDER BY
                id
        LIMIT   10
        OFFSET  90000
        ) q
JOIN    lookup l
ON      l.id = q.id
ORDER BY
        q.id

Query results

select length(`20110211_late`.`l`.`shorttxt`) AS `LENGTH(l.shorttxt)` from (select `20110211_late`.`lookup`.`id` AS `id` from `20110211_late`.`lookup` order by `20110211_late`.`lookup`.`id` limit 90000,10) `q` join `20110211_late`.`lookup` `l` where (`20110211_late`.`l`.`id` = `q`.`id`) order by `q`.`id`

No rewrite:

SELECT  LENGTH(shorttxt)
FROM    lookup
ORDER BY
        id
LIMIT   10
OFFSET  90000

Query results

select length(`20110211_late`.`lookup`.`shorttxt`) AS `LENGTH(shorttxt)` from `20110211_late`.`lookup` order by `20110211_late`.`lookup`.`id` limit 90000,10

There is quite a significant difference (92 ms vs 40 ms) the reasons for which we will discuss a little bit later, after we see the results of the third query.

PRIMARY KEY and the long TEXT column

Rewrite:

SELECT  LENGTH(l.longtxt)
FROM    (
        SELECT  id
        FROM    lookup
        ORDER BY
                id
        LIMIT   10
        OFFSET  90000
        ) q
JOIN    lookup l
ON      l.id = q.id
ORDER BY
        q.id

Query results

select length(`20110211_late`.`l`.`longtxt`) AS `LENGTH(l.longtxt)` from (select `20110211_late`.`lookup`.`id` AS `id` from `20110211_late`.`lookup` order by `20110211_late`.`lookup`.`id` limit 90000,10) `q` join `20110211_late`.`lookup` `l` where (`20110211_late`.`l`.`id` = `q`.`id`) order by `q`.`id`

No rewrite:

SELECT  LENGTH(longtxt)
FROM    lookup
ORDER BY
        id
LIMIT   10
OFFSET  90000

Query results

select length(`20110211_late`.`lookup`.`longtxt`) AS `LENGTH(longtxt)` from `20110211_late`.`lookup` order by `20110211_late`.`lookup`.`id` limit 90000,10

The query employing the trick runs for same 40 ms, while the straightforward query takes as much as 30 seconds (30,359 ms, to be exact).

Why such a difference?

The reason is that InnoDB, despite the fact it stores the data in the clustered index, is still able to move some data out of the index. This is called external storage.

With COMPACT row format I used to create the tables, InnoDB, when trying to insert a record with two TEXT columns on the page, will try to fit both of them on the page. If this is not possible, then it will split the longest of the records in two parts: the first 768 bytes will be stored on the page and the remaining data will be stored on a separate page (or pages), with a pointer to these data stored in the original clustered index. This will be repeated until all TEXT columns fit on the page of there is no more space there (in which case an error would be thrown).

This means that all TEXT columns shorter than 768 bytes will be stored completely on the page, while those longer can be either split of stored as a whole (with at least first 768 bytes still being on the page).

With the column lengths chosen (1 to 100 and 8K to 8K + 100, accordingly), it can be easily seen that shorttxt will always be stored on-page, while longtxt will always be split (since InnoDB allows at most 8K per record (minus some overhead) to be stored on one page).

Now, this becomes more clear. As with MyISAM, the straightforward query involving longtxt should perform two page lookups per each record scanned: the first one on the clustered index, the second one on the external storage. This takes lots of time and may even spoil the InnoDB cache with unneeded data (which would lead to increased cache miss ratio).

The query on shorttxt is not that bad, since it only requires come extra CPU cycles per record to calculate the string length.

Now, let’s check one more query which orders by a secondary indexed field rather than id:

Secondary index and the short text column

Rewrite:

SELECT  LENGTH(l.shorttxt)
FROM    (
        SELECT  id, value
        FROM    lookup
        ORDER BY
                value
        LIMIT   10
        OFFSET  90000
        ) q
JOIN    lookup l
ON      l.id = q.id
ORDER BY
        q.value

Query results

select length(`20110211_late`.`l`.`shorttxt`) AS `LENGTH(l.shorttxt)` from (select `20110211_late`.`lookup`.`id` AS `id`,`20110211_late`.`lookup`.`value` AS `value` from `20110211_late`.`lookup` order by `20110211_late`.`lookup`.`value` limit 90000,10) `q` join `20110211_late`.`lookup` `l` where (`20110211_late`.`l`.`id` = `q`.`id`) order by `q`.`value`

No rewrite:

SELECT  LENGTH(shorttxt)
FROM    lookup
ORDER BY
        value
LIMIT   10
OFFSET  90000

Query results

select length(`20110211_late`.`lookup`.`shorttxt`) AS `LENGTH(shorttxt)` from `20110211_late`.`lookup` order by `20110211_late`.`lookup`.`value` limit 90000,10

The execution times for these queries vary tenfold: 26 ms vs 266 ms.

As with MyISAM, the secondary index requires an extra lookup to retrieve the actual values from the table (the only difference is that any index is secondary in MyISAM, including that used to police the PRIMARY KEY).

The first query does not perform these row lookups on the skipped records and hence is ten times as fast. It is even faster than queries ordering on the PRIMARY KEY, because the secondary index contains significantly less data than the PRIMARY KEY, holds much more records per page, is hence more shallow and can be traversed faster.

The second query does perform the early row lookups, as usual.

Conclusion

A trick used to avoid early row lookups for the LIMIT … OFFSET queries is useful on InnoDB tables too, though to different extent, depending on the ORDER BY condition and the columns involved:

• It’s very useful on queries involving columns stored off-page (long TEXT, BLOB and VARCHAR columns)
• It’s very useful on ORDER BY conditions served by secondary indexes
• It’s quite useful on moderate sized columns (still stored on page) or CPU-intensive expressions
• It’s almost useless on short columns without complex CPU-intensive processing

Hope that helps.

I’m always glad to answer the questions regarding database queries.

Ask me a question

#10. Searching for a NULL

SELECT  *
FROM    a
WHERE   a.column = NULL

In SQL, a NULL is never equal to anything, even another NULL. This query won’t return anything and in fact will be thrown out by the optimizer when building the plan.

When searching for NULL values, use this instead:

SELECT  *
FROM    a
WHERE   a.column IS NULL

#9. LEFT JOIN with additional conditions

SELECT  *
FROM    a
LEFT JOIN
        b
ON      b.a = a.id
WHERE   b.column = 'something'

A LEFT JOIN is like INNER JOIN except that it will return each record from a at least once, substituting missing fields from b with NULL values, if there are no actual matching records.

The WHERE condition, however, is evaluated after the LEFT JOIN so the query above checks column after it had been joined. And as we learned earlier, no NULL value can satisfy an equality condition, so the records from a without corresponding record from b will unavoidably be filtered out.

Essentially, this query is an INNER JOIN, only less efficient.

To match only the records with b.column = 'something' (while still returning all records from a), this condition should be moved into ON clause:

SELECT  *
FROM    a
LEFT JOIN
        b
ON      b.a = a.id
        AND b.column = 'something'

#8. Less than a value but not a NULL

Quite often I see the queries like this:

SELECT  *
FROM    b
WHERE   b.column < 'something'
        AND b.column IS NOT NULL

This is actually not an error: this query is valid and will do what’s intended. However, IS NOT NULL here is redundant.

If b.column is a NULL, then b.column < 'something' will never be satisfied, since any comparison to NULL evaluates to a boolean NULL and does not pass the filter.

It is interesting that this additional NULL check is never used for greater than queries (like in b.column > 'something').

This is because NULL go first in ORDER BY in MySQL and hence are incorrectly considered less than any other value by some people.

This query can be simplified:

SELECT  *
FROM    b
WHERE   b.column < 'something'

and will still never return a NULL in b.column.

#7. Joining on NULL

SELECT  *
FROM    a
JOIN    b
ON      a.column = b.column

When column is nullable in both tables, this query won't return a match of two NULLs for the reasons described above: no NULLs are equal.

Here's a query to do that:

SELECT  *
FROM    a
JOIN    b
ON      a.column = b.column
        OR (a.column IS NULL AND b.column IS NULL)

MySQL's optimizer treats this as an equijoin and provides a special join condition, ref_or_null.

#6. NOT IN with NULL values

SELECT  a.*
FROM    a
WHERE   a.column NOT IN
        (
        SELECT column
        FROM    b
        )

This query will never return anything if there is but a single NULL in b.column. As with other predicates, both IN and NOT IN against NULL evaluate to NULL.

This should be rewritten using a NOT EXISTS:

SELECT  a.*
FROM    a
WHERE   NOT EXISTS
        (
        SELECT NULL
        FROM    b
        WHERE   b.column = a.column
        )

Unlike IN, EXISTS always evaluates to either true or false.

#5. Ordering random samples

SELECT  *
FROM    a
ORDER BY
        RAND(), column
LIMIT 10

This query attempts to select 10 random records ordered by column.

ORDER BY orders the output lexicographically: that is, the records are only ordered on the second expression when the values of the first expression are equal.

However, the results of RAND() are, well, random. It's infeasible that the values of RAND() will match, so ordering on column after RAND() is quite useless.

To order the randomly sampled records, use this query:

SELECT  *
FROM    (
        SELECT  *
        FROM    mytable
        ORDER BY
                RAND()
        LIMIT 10
        ) q
ORDER BY
        column

#4. Sampling arbitrary record from a group

This query intends to select one column from each group (defined by grouper)

SELECT  DISTINCT(grouper), a.*
FROM    a

DISTINCT is not a function, it's a part of SELECT clause. It applies to all columns in the SELECT list, and the parentheses here may just be omitted. This query may and will select the duplicates on grouper (if the values in at least one of the other columns differ).

Sometimes, it's worked around using this query (which relies on MySQL's extensions to GROUP BY):

SELECT  a.*
FROM    a
GROUP BY
        grouper

Unaggregated columns returned within each group are arbitrarily taken.

At first, this appears to be a nice solution, but it has quite a serious drawback. It relies on the assumption that all values returned, though taken arbitrarily from the group, will still belong to one record.

Though with current implementation is seems to be so, it's not documented and can be changed in any moment (especially if MySQL will ever learn to apply index_union after GROUP BY). So it's not safe to rely on this behavior.

This query would be easy to rewrite in a cleaner way if MySQL supported analytic functions. However, it's still possible to make do without them, if the table has a PRIMARY KEY defined:

SELECT  a.*
FROM    (
        SELECT  DISTINCT grouper
        FROM    a
        ) ao
JOIN    a
ON      a.id = 
        (
        SELECT  id
        FROM    a ai
        WHERE   ai.grouper = ao.grouper
        LIMIT 1
        )

#3. Sampling first record from a group

This is a variation of the previous query:

SELECT  a.*
FROM    a
GROUP BY
        grouper
ORDER BY
        MIN(id) DESC

Unlike the previous query, this one attempts to select the record holding the minimal id.

Again: it is not guaranteed that the unaggregated values returned by a.* will belong to a record holding MIN(id) (or even to a single record at all).

Here's how to do it in a clean way:

SELECT  a.*
FROM    (
        SELECT  DISTINCT grouper
        FROM    a
        ) ao
JOIN    a
ON      a.id = 
        (
        SELECT  id
        FROM    a ai
        WHERE   ai.grouper = ao.grouper
        ORDER BY
                ai.grouper, ai.id
        LIMIT 1
        )

This query is just like the previous one but with ORDER BY added to ensure that the first record in id order will be returned.

#2. IN and comma-separated list of values

This query attempts to match the value of column against any of those provided in a comma-separated string:

SELECT  *
FROM    a
WHERE   column IN ('1, 2, 3')

This does not work because the string is not expanded in the IN list.

Instead, if column column is a VARCHAR, it is compared (as a string) to the whole list (also as a string), and of course will never match. If column is of a numeric type, the list is cast into the numeric type as well (and only the first item will match, at best).

The correct way to deal with this query would be rewriting it as a proper IN list

SELECT  *
FROM    a
WHERE   column IN (1, 2, 3)

, or as an inline view:

SELECT  *
FROM    (
        SELECT  1 AS id
        UNION ALL
        SELECT  2 AS id
        UNION ALL
        SELECT  3 AS id
        ) q
JOIN    a
ON      a.column = q.id

, but this is not always possible.

To work around this without changing the query parameters, one can use FIND_IN_SET:

SELECT  *
FROM    a
WHERE   FIND_IN_SET(column, '1,2,3')

This function, however, is not sargable and a full table scan will be performed on a.

#1. LEFT JOIN with COUNT(*)

SELECT  a.id, COUNT(*)
FROM    a
LEFT JOIN
        b
ON      b.a = a.id
GROUP BY
        a.id

This query intends to count number of matches in b for each record in a.

The problem is that COUNT(*) will never return a 0 in such a query. If there is no match for a certain record in a, the record will be still returned and counted.

COUNT should be made to count only the actual records in b. Since COUNT(*), when called with an argument, ignores NULLs, we can pass b.a to it. As a join key, it can never be a null in an actual match, but will be if there were no match:

SELECT  a.id, COUNT(b.a)
FROM    a
LEFT JOIN
        b
ON      b.a = a.id
GROUP BY
        a.id

P.S. In case you were wondering: no, the pictures don't have any special meaning. I just liked them.

I have a table of blog posts, each with a foreign key back to it’s author. There are less than 15,000 entries in this table.

This query scans over 19,000 rows (per EXPLAIN), requires a filesort (that might be regular MySQL behavior), and takes over 400ms to return 5 rows. possibly because of the complicated WHERE used to check if the item is actually published.

SELECT  blog_post.id,
        blog_post.title,
        blog_post.author_id,
        blog_post.has_been_fact_checked,
        blog_post.published_date,
        blog_post.ordering,
        auth_user.username,
        auth_user.email
FROM    blog_post
INNER JOIN
        auth_user
ON      auth_user.id = blog_post.author_id
WHERE   blog_post.is_approved = 1
        AND blog_post.has_been_fact_checked = 1
        AND blog_post.published_date <=  '2010-10-25 22:40:05'
ORDER BY
        blog_post.published_date DESC,
        blog_post.ordering ASC,
        blog_post.id DESC
LIMIT 5

How can I wrangle this query under control?

This query is quite simple: a filtering condition with two equalities and a range and an order by. The range in the filter fits the ORDER BY perfectly, so the whole query could be executed using an index scan without any filesorts.

The only problem is that we have the mixed directions in ORDER BY, and MySQL does not support ASC / DESC clauses for the indexes.

With a simple table redesign, the problem could easily be solved: we would just need to reverse the order in the ordering column, say, by creating its copy and storing negative values in it. This way, we could just create a composite index (with all columns ascending) and rewrite the query to use the reverse column instead. That’s what many engines do, MediaWiki (which Wikipedia runs on) being one of the most well-known examples.

This solution is nice, I hear people saying, but requires a database redesign which as we all know is never as simple as some development pundits on their blogs seem to think.

OK, this is a good point. Let’s see what we can do with the current design, and, as always, create a sample table to do it:

Table creation details

CREATE TABLE filler (
        id INT NOT NULL PRIMARY KEY AUTO_INCREMENT
) ENGINE=Memory;

CREATE TABLE blog_post
        (
        id INT NOT NULL PRIMARY KEY,
        title TEXT NOT NULL,
        is_approved BOOLEAN NOT NULL,
        has_been_fact_checked BOOLEAN NOT NULL,
        published_date DATETIME,
        ordering INT,
        author_id INT NOT NULL,
        KEY ix_blogpost_approved_checked_published_ordering_id
                (
                is_approved,
                has_been_fact_checked,
                published_date,
                ordering
                )
        )
ENGINE=InnoDB;

CREATE TABLE auth_user
        (
        id INT NOT NULL PRIMARY KEY, 
        username VARCHAR(32) NOT NULL,
        email VARCHAR(256) NOT NULL
        )
ENGINE=InnoDB;

DELIMITER $$

CREATE PROCEDURE prc_filler(cnt INT)
BEGIN
        DECLARE _cnt INT;
        SET _cnt = 1;
        WHILE _cnt <= cnt DO
                INSERT
                INTO    filler
                SELECT  _cnt;
                SET _cnt = _cnt + 1;
        END WHILE;
END
$$

DELIMITER ;

START TRANSACTION;
CALL prc_filler(500000);
COMMIT;

INSERT
INTO    auth_user
SELECT  id, CONCAT('author', id), CONCAT('author', id, '@example.com')
FROM    filler
LIMIT 1000;

INSERT
INTO    blog_post
SELECT  id, CONCAT('Post ', id),
        RAND(20101102) > 0.05,
        RAND(20101102 << 1) > 0.05,
        CAST('2010-11-02' AS DATE) - INTERVAL RAND(20101102 << 2) * 100000 HOUR,
        CEILING(RAND(20101102 << 3) * 100),
        CEILING(RAND(20101102 << 4) * 1000)
FROM    filler;

There are 1,000 authors and 500,000 blog posts in the respective tables. The blog posts intentionally have duplicates on published_date.

The original query:

SELECT  blog_post.id,
        blog_post.title,
        blog_post.author_id,
        blog_post.has_been_fact_checked,
        blog_post.published_date,
        blog_post.ordering,
        auth_user.username,
        auth_user.email
FROM    blog_post
INNER JOIN
        auth_user
ON      auth_user.id = blog_post.author_id
WHERE   blog_post.is_approved = 1
        AND blog_post.has_been_fact_checked = 1
        AND blog_post.published_date <= '2010-10-25 22:40:05'
ORDER BY
        blog_post.published_date DESC,
        blog_post.ordering ASC,
        blog_post.id DESC
LIMIT 5

select `20101102_desc`.`blog_post`.`id` AS `id`,`20101102_desc`.`blog_post`.`title` AS `title`,`20101102_desc`.`blog_post`.`author_id` AS `author_id`,`20101102_desc`.`blog_post`.`has_been_fact_checked` AS `has_been_fact_checked`,`20101102_desc`.`blog_post`.`published_date` AS `published_date`,`20101102_desc`.`blog_post`.`ordering` AS `ordering`,`20101102_desc`.`auth_user`.`username` AS `username`,`20101102_desc`.`auth_user`.`email` AS `email` from `20101102_desc`.`blog_post` join `20101102_desc`.`auth_user` where ((`20101102_desc`.`auth_user`.`id` = `20101102_desc`.`blog_post`.`author_id`) and (`20101102_desc`.`blog_post`.`has_been_fact_checked` = 1) and (`20101102_desc`.`blog_post`.`is_approved` = 1) and (`20101102_desc`.`blog_post`.`published_date` <= '2010-10-25 22:40:05')) order by `20101102_desc`.`blog_post`.`published_date` desc,`20101102_desc`.`blog_post`.`ordering`,`20101102_desc`.`blog_post`.`id` desc limit 5

As the asker said, this query takes 455 ms and the filesort is used.

Now, as we cannot use a single index on mixed ASC / DESC ordering conditions, we should squeeze the most from our existing index.

The longest common prefix of the index and the ORDER BY expression (which implicitly includes all columns from the WHERE clause) is (is_approved, has_been_fact_checked, published_date).

What does it mean? It means that the chunks of data sharing the distinct values of these three columns will always go in the same order both in the index and in the query, though the order may and will differ within each chunk.

See the queries below, first one with the initial ORDER BY sorting:

SELECT  published_date, id, ordering
FROM    blog_post
WHERE   is_approved = 1
        AND has_been_fact_checked = 1
ORDER BY
        published_date DESC,
        ordering ASC,
        id DESC
LIMIT 16

, the second one with the index sorting:

SELECT  published_date, id, ordering
FROM    blog_post
WHERE   is_approved = 1
        AND has_been_fact_checked = 1
ORDER BY
        published_date DESC,
        ordering DESC,
        id DESC
LIMIT 16

As you can see, the chunks of data occupy the same places within both recordsets. Post dated 2010-11-02 00:00:00 is at row 1. Those dated 2010-11-01 23:00:00 are at rows 2 — 10 etc. The orders differ only within the chunks, but not between chunks.

This means that if we select the same chunks as the original query does and reorder them, we get the same order.

And the chunks we can select quite efficiently using the index, since their order, as we shown above, is the same for both queries.

The original query had LIMIT 5, this means we have to select at most 5 chunks. To do this, we just need to count off 5 distinct dates satisfying the condition.

This can be done using a technique I described in one of my earlier posts:

• 20 latest unique records

Selecting 5 dates

Here’s a query to select 5 distinct dates less than the given one:

SELECT  bp.published_date
FROM    blog_post bp
WHERE   bp.is_approved = 1
        AND bp.has_been_fact_checked = 1
        AND bp.published_date < '2010-10-25 22:40:05'
        AND bp.id =
        (
        SELECT  id
        FROM    blog_post bpi
        WHERE   bpi.is_approved = 1
                AND bpi.has_been_fact_checked = 1
                AND bpi.published_date = bp.published_date
        ORDER BY
                bpi.is_approved DESC, bpi.has_been_fact_checked DESC, bpi.published_date DESC
        LIMIT 1
        )
ORDER BY
        bp.is_approved DESC, bp.has_been_fact_checked DESC, bp.published_date DESC
LIMIT 5

Field or reference '20101102_desc.bp.published_date' of SELECT #2 was resolved in SELECT #1
select `20101102_desc`.`bp`.`published_date` AS `published_date` from `20101102_desc`.`blog_post` `bp` where ((`20101102_desc`.`bp`.`has_been_fact_checked` = 1) and (`20101102_desc`.`bp`.`is_approved` = 1) and (`20101102_desc`.`bp`.`published_date` < '2010-10-25 22:40:05') and (`20101102_desc`.`bp`.`id` = (select `20101102_desc`.`bpi`.`id` from `20101102_desc`.`blog_post` `bpi` where ((`20101102_desc`.`bpi`.`has_been_fact_checked` = 1) and (`20101102_desc`.`bpi`.`is_approved` = 1) and (`20101102_desc`.`bpi`.`published_date` = `20101102_desc`.`bp`.`published_date`)) order by `20101102_desc`.`bpi`.`is_approved` desc,`20101102_desc`.`bpi`.`has_been_fact_checked` desc,`20101102_desc`.`bpi`.`published_date` desc limit 1))) order by `20101102_desc`.`bp`.`is_approved` desc,`20101102_desc`.`bp`.`has_been_fact_checked` desc,`20101102_desc`.`bp`.`published_date` desc limit 5

This uses the index access and is very efficient.

Selecting chunks of data

Now we need to select all records within the chunks defined by the dates. This can be done with a simple BETWEEN condition. The upper bound of the BETWEEN will be the date that we used as the parameter to the query, and the lower one will be the last date of those selected on the previous stage.

Since it should be a scalar, we will just replace LIMIT 5 (which returns 5 records) with LIMIT 4, 1 (which returns the last of these 5 records) and put it into a scalar subquery:

SELECT  *
FROM    blog_post bpo
WHERE   is_approved = 1
        AND has_been_fact_checked = 1
        AND published_date BETWEEN
        (
        SELECT  bp.published_date
        FROM    blog_post bp
        WHERE   bp.is_approved = 1
                AND bp.has_been_fact_checked = 1
                AND bp.published_date < '2010-10-25 22:40:05'
                AND bp.id =
                (
                SELECT  id
                FROM    blog_post bpi
                WHERE   bpi.is_approved = 1
                        AND bpi.has_been_fact_checked = 1
                        AND bpi.published_date = bp.published_date
                ORDER BY
                        bpi.is_approved DESC, bpi.has_been_fact_checked DESC, bpi.published_date DESC
                LIMIT 1
                )
        ORDER BY
                bp.is_approved DESC, bp.has_been_fact_checked DESC, bp.published_date DESC
        LIMIT 4, 1
        )
        AND '2010-10-25 22:40:05'

Field or reference '20101102_desc.bp.published_date' of SELECT #3 was resolved in SELECT #2
select `20101102_desc`.`bpo`.`id` AS `id`,`20101102_desc`.`bpo`.`title` AS `title`,`20101102_desc`.`bpo`.`is_approved` AS `is_approved`,`20101102_desc`.`bpo`.`has_been_fact_checked` AS `has_been_fact_checked`,`20101102_desc`.`bpo`.`published_date` AS `published_date`,`20101102_desc`.`bpo`.`ordering` AS `ordering`,`20101102_desc`.`bpo`.`author_id` AS `author_id` from `20101102_desc`.`blog_post` `bpo` where ((`20101102_desc`.`bpo`.`has_been_fact_checked` = 1) and (`20101102_desc`.`bpo`.`is_approved` = 1) and (`20101102_desc`.`bpo`.`published_date` between (select `20101102_desc`.`bp`.`published_date` from `20101102_desc`.`blog_post` `bp` where ((`20101102_desc`.`bp`.`has_been_fact_checked` = 1) and (`20101102_desc`.`bp`.`is_approved` = 1) and (`20101102_desc`.`bp`.`published_date` < '2010-10-25 22:40:05') and (`20101102_desc`.`bp`.`id` = (select `20101102_desc`.`bpi`.`id` from `20101102_desc`.`blog_post` `bpi` where ((`20101102_desc`.`bpi`.`has_been_fact_checked` = 1) and (`20101102_desc`.`bpi`.`is_approved` = 1) and (`20101102_desc`.`bpi`.`published_date` = `20101102_desc`.`bp`.`published_date`)) order by `20101102_desc`.`bpi`.`is_approved` desc,`20101102_desc`.`bpi`.`has_been_fact_checked` desc,`20101102_desc`.`bpi`.`published_date` desc limit 1))) order by `20101102_desc`.`bp`.`is_approved` desc,`20101102_desc`.`bp`.`has_been_fact_checked` desc,`20101102_desc`.`bp`.`published_date` desc limit 4,1) and '2010-10-25 22:40:05'))

This query returns (almost instantly) 20 records in 5 chunks of different size (split for readability). Note that the chunks are arranged in order reverse to the order the dates were returned on the previous step: we omitted the ORDER BY clause and, by default, the index is traversed in forward (ASC) direction.

Ordering and joining

Now, all that’s left to do is to add the correct ORDER BY condition to rearrange the data within the chunks, join the authors table and apply the LIMIT:

SELECT  bpo.id,
        bpo.title,
        bpo.author_id,
        bpo.has_been_fact_checked,
        bpo.published_date,
        bpo.ordering,
        au.username,
        au.email
FROM    blog_post bpo
JOIN    auth_user au
ON      au.id = bpo.author_id
WHERE   bpo.is_approved = 1
        AND bpo.has_been_fact_checked = 1
        AND bpo.published_date BETWEEN
        (
        SELECT  bp.published_date
        FROM    blog_post bp
        WHERE   bp.is_approved = 1
                AND bp.has_been_fact_checked = 1
                AND bp.published_date < '2010-10-25 22:40:05'
                AND bp.id =
                (
                SELECT  id
                FROM    blog_post bpi
                WHERE   bpi.is_approved = 1
                        AND bpi.has_been_fact_checked = 1
                        AND bpi.published_date = bp.published_date
                ORDER BY
                        bpi.is_approved DESC, bpi.has_been_fact_checked DESC, bpi.published_date DESC
                LIMIT 1
                )
        ORDER BY
                bp.is_approved DESC, bp.has_been_fact_checked DESC, bp.published_date DESC
        LIMIT 4, 1
        )
        AND '2010-10-25 22:40:05'
ORDER BY
        bpo.published_date DESC,
        bpo.ordering ASC,
        bpo.id DESC
LIMIT 5

Field or reference '20101102_desc.bp.published_date' of SELECT #3 was resolved in SELECT #2
select `20101102_desc`.`bpo`.`id` AS `id`,`20101102_desc`.`bpo`.`title` AS `title`,`20101102_desc`.`bpo`.`author_id` AS `author_id`,`20101102_desc`.`bpo`.`has_been_fact_checked` AS `has_been_fact_checked`,`20101102_desc`.`bpo`.`published_date` AS `published_date`,`20101102_desc`.`bpo`.`ordering` AS `ordering`,`20101102_desc`.`au`.`username` AS `username`,`20101102_desc`.`au`.`email` AS `email` from `20101102_desc`.`blog_post` `bpo` join `20101102_desc`.`auth_user` `au` where ((`20101102_desc`.`au`.`id` = `20101102_desc`.`bpo`.`author_id`) and (`20101102_desc`.`bpo`.`has_been_fact_checked` = 1) and (`20101102_desc`.`bpo`.`is_approved` = 1) and (`20101102_desc`.`bpo`.`published_date` between (select `20101102_desc`.`bp`.`published_date` from `20101102_desc`.`blog_post` `bp` where ((`20101102_desc`.`bp`.`has_been_fact_checked` = 1) and (`20101102_desc`.`bp`.`is_approved` = 1) and (`20101102_desc`.`bp`.`published_date` < '2010-10-25 22:40:05') and (`20101102_desc`.`bp`.`id` = (select `20101102_desc`.`bpi`.`id` from `20101102_desc`.`blog_post` `bpi` where ((`20101102_desc`.`bpi`.`has_been_fact_checked` = 1) and (`20101102_desc`.`bpi`.`is_approved` = 1) and (`20101102_desc`.`bpi`.`published_date` = `20101102_desc`.`bp`.`published_date`)) order by `20101102_desc`.`bpi`.`is_approved` desc,`20101102_desc`.`bpi`.`has_been_fact_checked` desc,`20101102_desc`.`bpi`.`published_date` desc limit 1))) order by `20101102_desc`.`bp`.`is_approved` desc,`20101102_desc`.`bp`.`has_been_fact_checked` desc,`20101102_desc`.`bp`.`published_date` desc limit 4,1) and '2010-10-25 22:40:05')) order by `20101102_desc`.`bpo`.`published_date` desc,`20101102_desc`.`bpo`.`ordering`,`20101102_desc`.`bpo`.`id` desc limit 5

The records have been rearranged (using a filesort) and a LIMIT was applied.

Since the records are only taken from 5 chunks with 20 records in total, the filesort is not a problem at all and the query completes in only 4 ms.

Conclusion

MySQL does not support ASC / DESC clauses in the indices.

However, with an ORDER BY / LIMIT condition that mixes the orders of columns, it is possible to use an index.

To do that, we should limit the recordset to the minimal set that would still retain the order on the leading columns that share one direction. This can be done by selecting distinct set of those leading columns up to the original limit, and filtering on them using the index.

Then we can revert to a filesort to order this (much smaller) set on the remaining columns, which would be much more efficient.

Beren asks:

In my database, I store the information about the game matches.

A match can be played between two players (mostly) or, occasionally, between two teams of 2, 3 or 4 players.

I store the matches in a table like this:

Game

id	player1	player2	type

, where type is ENUM(player, team).

If type is player, the ids of players are stored in the record; if type is team, those of teams are stored.

Now, the tricky part. For a given game, I need to select two lists of players from both sides, comma separated, be they single players or members of a team.

Are two separate queries required for this?

This of course can be easily done using a single query.

Let’s make a sample table and see:

Table creation details

CREATE TABLE filler (
        id INT NOT NULL PRIMARY KEY AUTO_INCREMENT
) ENGINE=Memory;

CREATE TABLE game
        (
        id INT NOT NULL PRIMARY KEY,
        name VARCHAR(30) NOT NULL,
        player1 INT NOT NULL,
        player2 INT NOT NULL,
        type ENUM('player', 'team')
        )
ENGINE=InnoDB;

CREATE TABLE player
        (
        id INT NOT NULL PRIMARY KEY,
        name VARCHAR(30) NOT NULL
        )
ENGINE=InnoDB;

CREATE TABLE team
        (
        id INT NOT NULL PRIMARY KEY,
        name VARCHAR(30) NOT NULL
        )
ENGINE=InnoDB;

CREATE TABLE player_team
        (
        team INT NOT NULL,
        player INT NOT NULL,
        PRIMARY KEY (team, player)
        )
ENGINE=InnoDB;

DELIMITER $$

CREATE PROCEDURE prc_filler(cnt INT)
BEGIN
        DECLARE _cnt INT;
        SET _cnt = 1;
        WHILE _cnt <= cnt DO
                INSERT
                INTO    filler
                SELECT  _cnt;
                SET _cnt = _cnt + 1;
        END WHILE;
END
$$

DELIMITER ;

START TRANSACTION;
CALL prc_filler(500000);
COMMIT;

INSERT
INTO    player
SELECT  id, CONCAT('Player ', id)
FROM    filler
ORDER BY
        id
LIMIT 100000;

INSERT
INTO    team
SELECT  id, CONCAT('Team ', id)
FROM    filler
ORDER BY
        id
LIMIT 3000;

INSERT
INTO    player_team
SELECT  t.id,
        FLOOR(RAND(20101020) * 25000) * 4 + num
FROM    team t
JOIN    (
        SELECT  1 AS num
        UNION ALL
        SELECT  2
        UNION ALL
        SELECT  3
        UNION ALL
        SELECT  4
        ) n
ON      n.num <= ((t.id - 1) % 3) + 2;

INSERT
INTO    game
SELECT  id, CONCAT('Game ', id),
        CASE n
        WHEN 1 THEN
                CEILING(RAND(20101020 << 2) * 100000)
        ELSE
                FLOOR(RAND(20101020 << 3) * 1000) * 3 + n - 1
        END,
        CASE n
        WHEN 1 THEN
                CEILING(RAND(20101020 << 4) * 100000)
        ELSE
                FLOOR(RAND(20101020 << 5) * 1000) * 3 + n - 1
        END,
        CASE n
        WHEN 1 THEN
                'player'
        ELSE
                'team'
        END
FROM    (
        SELECT  id, -FLOOR(LOG10(POWER(30, -4) + RAND(20100930) * (1 - POWER(30, -4))) / LOG10(30)) AS n
        FROM    filler
        ) q;

The tables contain 500,000 games, 100,000 players and 3,000 teams.

The participants in the games are distributed logarithmically: 95% of all games are single player, 95% of the remaining games are played by 2×2 teams, etc.:

The model above may seem quite weird, since the single players can be assigned to the fake teams so that the joins will be always performed through the player_team table. However, it’s not required and in fact is less efficient. We don’t really need that extra lookup if we can find a player right away, using its id, and since the majority of games are single player, majority of queries will be too.

Let’s see how we can build a list of players’ names. First, let’s do it for the player1 part only:

SELECT  g.id, g.name, GROUP_CONCAT(p.name SEPARATOR ', ') AS players
FROM    game g
LEFT JOIN
        player_team pt
ON      g.type = 'team'
        AND pt.team = g.player1
JOIN    player p
ON      p.id = CASE g.type WHEN 'player' THEN g.player1 WHEN 'team' THEN pt.player END
GROUP BY
        g.id
LIMIT 50

select `20101020_matches`.`g`.`id` AS `id`,`20101020_matches`.`g`.`name` AS `name`,group_concat(`20101020_matches`.`p`.`name` separator ', ') AS `players` from `20101020_matches`.`game` `g` left join `20101020_matches`.`player_team` `pt` on(((`20101020_matches`.`pt`.`team` = `20101020_matches`.`g`.`player1`) and (`20101020_matches`.`g`.`type` = 'team'))) join `20101020_matches`.`player` `p` where (`20101020_matches`.`p`.`id` = coalesce(`20101020_matches`.`pt`.`player`,`20101020_matches`.`g`.`player1`)) group by `20101020_matches`.`g`.`id` limit 50

We see that the query returns the lists of players correctly both for single-player games and team ones (like the one you can see at Game 38). The query works in the following way:

• For each record from game, it determines whether the game is single player or a team match.
• If the game is multiplayer, it joins the player_team table. The query is written so that the game type is a part of a LEFT JOIN condition which can be short-circuited by the optimizer. This means that if a game is a single player game, no actual lookup is performed against player_team: the optimizer will just optimize away this step, knowing for sure that it just needs to return a NULL.
• Finally, the player table is joined, using CASE. The case will evaluate to either player_team.player or to game.player1 (depending on the game type).

Theoretically, it could be replaced with a mere COALESCE(player_team.player, game.player1); however, if the game type were team but the team itself were empty, this expression would incorrectly evaluate to game.player1, not regarding the game type. So it’s better to leave it as is.

Now, we have an aggregate of the players’ names from the one side. How do we make the same for two sides?

Since we need to calculate aggregates for two different columns, it’s better to replace the JOINs with the correlated subqueries. However, there is a little problem.

Normally, a correlated subquery just replaces the ON condition with a WHERE condition. But this is with a plain JOIN, and we here have a LEFT JOIN. We cannot just replace it with WHERE: it won’t match anything.

Unfortunately, MySQL has a very limited ability of correlating the subqueries. For instance, a correlated value cannot be nested more than one level deep, neither can is be used in an ON clause of a join. That’s why we will need to make one extra join in our subqueries.

To rewrite the aggregate query as subqueries, we will need to join game to itselft in the subqueries and use the values of a joined table instead of the correlated values:

SELECT  g.id, g.name,
        (
        SELECT  GROUP_CONCAT(p.name SEPARATOR ', ') AS players
        FROM    game gi
        LEFT JOIN  
                player_team pt
        ON      gi.type = 'team'
                AND pt.team = gi.player1
        JOIN    player p
        ON      p.id = CASE gi.type WHEN 'player' THEN gi.player1 WHEN 'team' THEN pt.player END
        WHERE   gi.id = g.id
        ),
        (
        SELECT  GROUP_CONCAT(p.name SEPARATOR ', ') AS players
        FROM    game gi
        LEFT JOIN
                player_team pt
        ON      gi.type = 'team'
                AND pt.team = gi.player2
        JOIN    player p
        ON      p.id = CASE gi.type WHEN 'player' THEN gi.player2 WHEN 'team' THEN pt.player END
        WHERE   gi.id = g.id
        )
FROM    game g
ORDER BY
        id
LIMIT 50

Field or reference '20101020_matches.g.id' of SELECT #2 was resolved in SELECT #1
Field or reference '20101020_matches.g.id' of SELECT #3 was resolved in SELECT #1
select `20101020_matches`.`g`.`id` AS `id`,`20101020_matches`.`g`.`name` AS `name`,(select group_concat(`20101020_matches`.`p`.`name` separator ', ') AS `players` from `20101020_matches`.`game` `gi` left join `20101020_matches`.`player_team` `pt` on(((`20101020_matches`.`pt`.`team` = `20101020_matches`.`gi`.`player1`) and (`20101020_matches`.`gi`.`type` = 'team'))) join `20101020_matches`.`player` `p` where ((`20101020_matches`.`gi`.`id` = `20101020_matches`.`g`.`id`) and (`20101020_matches`.`p`.`id` = (case `20101020_matches`.`gi`.`type` when 'player' then `20101020_matches`.`gi`.`player1` when 'team' then `20101020_matches`.`pt`.`player` end)))) AS `(
        SELECT  GROUP_CONCAT(p.name SEPARATOR ', ') AS players
        FROM    game gi
        LEFT JOIN  
                player_team pt
        ON      gi.type = 'team'
                AND pt.team = gi.player1
        JOIN    player p
        ON      `,(select group_concat(`20101020_matches`.`p`.`name` separator ', ') AS `players` from `20101020_matches`.`game` `gi` left join `20101020_matches`.`player_team` `pt` on(((`20101020_matches`.`pt`.`team` = `20101020_matches`.`gi`.`player2`) and (`20101020_matches`.`gi`.`type` = 'team'))) join `20101020_matches`.`player` `p` where ((`20101020_matches`.`gi`.`id` = `20101020_matches`.`g`.`id`) and (`20101020_matches`.`p`.`id` = (case `20101020_matches`.`gi`.`type` when 'player' then `20101020_matches`.`gi`.`player2` when 'team' then `20101020_matches`.`pt`.`player` end)))) AS `(
        SELECT  GROUP_CONCAT(p.name SEPARATOR ', ') AS players
        FROM    game gi
        LEFT JOIN
                player_team pt
        ON      gi.type = 'team'
                AND pt.team = gi.player2
        JOIN    player p
        ON      p.` from `20101020_matches`.`game` `g` order by `20101020_matches`.`g`.`id` limit 50

Instead of using correlated values g.type and g.player*, we join another instance of game (aliased as gi) on g.id and use its fields in the join instead.

This of course adds some overhead to the query, however, the PRIMARY KEY join on an already cached record is quite fast.

Let’s run two queries that select player lists for all games and compare the execution times.

First, the grouping query:

SELECT  SUM(LENGTH(players))
FROM    (
        SELECT  g.id, g.name, GROUP_CONCAT(p.name SEPARATOR ', ') AS players
        FROM    game g
        LEFT JOIN
                player_team pt
        ON      g.type = 'team'
                AND pt.team = g.player1
        JOIN    player p
        ON      p.id = CASE g.type WHEN 'player' THEN g.player1 WHEN 'team' THEN pt.player END
        GROUP BY
                g.id
        ORDER BY
                NULL
        ) q

Show query details

select sum(length(`q`.`players`)) AS `SUM(LENGTH(players))` from (select `20101020_matches`.`g`.`id` AS `id`,`20101020_matches`.`g`.`name` AS `name`,group_concat(`20101020_matches`.`p`.`name` separator ', ') AS `players` from `20101020_matches`.`game` `g` left join `20101020_matches`.`player_team` `pt` on(((`20101020_matches`.`pt`.`team` = `20101020_matches`.`g`.`player1`) and (`20101020_matches`.`g`.`type` = 'team'))) join `20101020_matches`.`player` `p` where (`20101020_matches`.`p`.`id` = (case `20101020_matches`.`g`.`type` when 'player' then `20101020_matches`.`g`.`player1` when 'team' then `20101020_matches`.`pt`.`player` end)) group by `20101020_matches`.`g`.`id` order by NULL) `q`

And the subqueries:

SELECT  g.id, g.name,
        SUM(LENGTH(
        (
        SELECT  GROUP_CONCAT(p.name SEPARATOR ', ') AS players
        FROM    game gi
        LEFT JOIN  
                player_team pt
        ON      gi.type = 'team'
                AND pt.team = gi.player1
        JOIN    player p
        ON      p.id = CASE gi.type WHEN 'player' THEN gi.player1 WHEN 'team' THEN pt.player END
        WHERE   gi.id = g.id
        )
        )) AS len1, 
        SUM(LENGTH(
        (
        SELECT  GROUP_CONCAT(p.name SEPARATOR ', ') AS players
        FROM    game gi
        LEFT JOIN
                player_team pt
        ON      gi.type = 'team'
                AND pt.team = gi.player2
        JOIN    player p
        ON      p.id = CASE gi.type WHEN 'player' THEN gi.player2 WHEN 'team' THEN pt.player END
        WHERE   gi.id = g.id
        )
        )) AS len2
FROM    game g

Show query details

Field or reference '20101020_matches.g.id' of SELECT #2 was resolved in SELECT #1
Field or reference '20101020_matches.g.id' of SELECT #3 was resolved in SELECT #1
select `20101020_matches`.`g`.`id` AS `id`,`20101020_matches`.`g`.`name` AS `name`,sum(length((select group_concat(`20101020_matches`.`p`.`name` separator ', ') AS `players` from `20101020_matches`.`game` `gi` left join `20101020_matches`.`player_team` `pt` on(((`20101020_matches`.`pt`.`team` = `20101020_matches`.`gi`.`player1`) and (`20101020_matches`.`gi`.`type` = 'team'))) join `20101020_matches`.`player` `p` where ((`20101020_matches`.`gi`.`id` = `20101020_matches`.`g`.`id`) and (`20101020_matches`.`p`.`id` = (case `20101020_matches`.`gi`.`type` when 'player' then `20101020_matches`.`gi`.`player1` when 'team' then `20101020_matches`.`pt`.`player` end)))))) AS `len1`,sum(length((select group_concat(`20101020_matches`.`p`.`name` separator ', ') AS `players` from `20101020_matches`.`game` `gi` left join `20101020_matches`.`player_team` `pt` on(((`20101020_matches`.`pt`.`team` = `20101020_matches`.`gi`.`player2`) and (`20101020_matches`.`gi`.`type` = 'team'))) join `20101020_matches`.`player` `p` where ((`20101020_matches`.`gi`.`id` = `20101020_matches`.`g`.`id`) and (`20101020_matches`.`p`.`id` = (case `20101020_matches`.`gi`.`type` when 'player' then `20101020_matches`.`gi`.`player2` when 'team' then `20101020_matches`.`pt`.`player` end)))))) AS `len2` from `20101020_matches`.`game` `g`

The second query is only 40% longer, but it parses and returns twice as much data.

That means that it’s more efficient to run a single query to get both lists of players than to run a grouping query twice.

Hope that helps.

I’m always glad to answer the questions regarding database queries.

Ask me a question

Denis Kuzmenok asks:

I need some help with a query I’m fighting with.

I have these tables:

Product

id	parent	name

Site

id	region	name

Price

id	product	site	value

For the products with a certain product.parent, I need to select minimal and maximal price within the given site.region, and return first 10 products ordered by the minimal price.

Each parent has 100 to 10,000 products, and there are 10,000,000 records in price table.

We have a classical OLAP task here: a fact table (price) and two dimension tables (product and site).

The task would be almost trivial if we were given the exact values of product and site. This way, we could build a composite index on these fields and value, filter on the exact values and get the first 10 values from the index.

However, we are given not the values of the dimensions themselves but those of the higher levels (parent for product and region for site).

Since the values of the levels are not stored in the facts table, we cannot index them. We need to join the dimension tables and filter on them.

This article describes four ways to join the tables, their efficiency varying depending on the density of the dimension values.

Since the only algorithm to make the joins MySQL is capable of is nested loops, basically, we need to define the order in which the tables would be joined.

There are three tables and, hence, 3! = 6 permutations that define all possible join orders:

However, two dimension tables are completely independent on each other. This means that if they come one after another in the join their order does not actually matter: they both will be searched for independent values. This reduces the number of actual combinations:

The joins must be designed so that the tables with most selective conditions go first. This means that the join order is determined by the density of the values satisfying the criteria. The more values from the table satisfy the search criteria, the later should the table come in the join, so that by the time the join occurs, most values would have been already sifted out.

Now, let’s build the queries for all types of the join. To do this, we will create sample tables:

Table creation details

CREATE TABLE filler (
        id INT NOT NULL PRIMARY KEY AUTO_INCREMENT
) ENGINE=Memory;

CREATE TABLE product (
        id INT NOT NULL PRIMARY KEY,
        parent INT NOT NULL,
        name TEXT NOT NULL
) ENGINE=InnoDB;

CREATE TABLE price (
        id INT NOT NULL PRIMARY KEY,
        product INT NOT NULL,
        site INT NOT NULL,
        value FLOAT NOT NULL
) ENGINE=InnoDB;

CREATE TABLE site (
       id INT NOT NULL PRIMARY KEY,
       region INT NOT NULL,
       name TEXT NOT NULL
) ENGINE=InnoDB;

DELIMITER $$

CREATE PROCEDURE prc_filler(cnt INT)
BEGIN
        DECLARE _cnt INT;
        SET _cnt = 1;
        WHILE _cnt <= cnt DO
                INSERT
                INTO    filler
                SELECT  _cnt;
                SET _cnt = _cnt + 1;
        END WHILE;
END
$$

DELIMITER ;

START TRANSACTION;
CALL prc_filler(250000);
COMMIT;

INSERT
INTO    product
SELECT  id, -FLOOR(LOG10(POWER(10, -10) + RAND(20100930) * (1 - POWER(10, -10))) * 500 / 10), CONCAT('Product ', id)
FROM    filler
ORDER BY
        id
LIMIT 250000;

INSERT
INTO    site
SELECT  id, -FLOOR(LOG10(POWER(10, -10) + RAND(20100930) * (1 - POWER(10, -10))) * 100 / 10), CONCAT('Site ', id)
FROM    filler
LIMIT 50000;

SET @r := 0;
INSERT
INTO    price
SELECT  @r := @r + 1,
        FLOOR(RAND(20100930 << 2) * 250000) + 1,
        FLOOR(RAND(20100930 << 3) * 50000) + 1,
        (RAND(20100930 << 4) * 100) + 20
FROM    product
CROSS JOIN
        (
        SELECT  id
        FROM    filler
        LIMIT 25
        ) q;

CREATE INDEX ix_product_parent ON product (parent);
CREATE INDEX ix_site_region ON site (region);
CREATE INDEX ix_price_site_product_value_id ON price (site, product, value, id);
CREATE INDEX ix_price_product_value_id ON price (product, value, id);
CREATE INDEX ix_price_value_id ON price (value, id);

Members of the dimensions are distributed logarithmically, reducing from the lower to the higher values.

Say, 11,379 products of 250,000 (4.55% of all products) belong to the parent 1, while only 2,906 products (1.16%) belong to the parent 30:

Same distribution applies to the sites:

Product/site, price

This is the most straightforward query. Since the product and site table come first, it’s best for sparse products and sparse sites.

They will be filtered first and the results of the filtering cross joined (still returning relatively few values to join against the prices).

Here’s the query:

SELECT  p.*, MIN(r.value) AS min_value
FROM    product p
CROSS JOIN
        site s
STRAIGHT_JOIN
        price r
ON      r.product = p.id
        AND r.site = s.id
WHERE   p.parent = 100
        AND s.region = 30
GROUP BY
        p.id
ORDER BY
        min_value
LIMIT 10

select `20100930_prices`.`p`.`id` AS `id`,`20100930_prices`.`p`.`parent` AS `parent`,`20100930_prices`.`p`.`name` AS `name`,min(`20100930_prices`.`r`.`value`) AS `min_value` from `20100930_prices`.`product` `p` join `20100930_prices`.`site` `s` straight_join `20100930_prices`.`price` `r` where ((`20100930_prices`.`r`.`site` = `20100930_prices`.`s`.`id`) and (`20100930_prices`.`r`.`product` = `20100930_prices`.`p`.`id`) and (`20100930_prices`.`s`.`region` = 30) and (`20100930_prices`.`p`.`parent` = 100)) group by `20100930_prices`.`p`.`id` order by min(`20100930_prices`.`r`.`value`) limit 10

We see that there is a filesort operation in the plan, but since there are relatively few values to sort, this operation is quite fast.

This query completes in 13 ms.

Site, product, price

This query is quite similar to the previous one, differing only in the JOIN order. It is best for sparse sites and dense products:

SELECT  p.*, MIN(r.value) AS min_value
FROM    site s
STRAIGHT_JOIN
        price r
ON      r.site = s.id
STRAIGHT_JOIN
        product p
ON      p.parent = 1
        AND p.id = r.product
WHERE   s.region = 30
GROUP BY
        p.id
ORDER BY
        min_value
LIMIT 10

select `20100930_prices`.`p`.`id` AS `id`,`20100930_prices`.`p`.`parent` AS `parent`,`20100930_prices`.`p`.`name` AS `name`,min(`20100930_prices`.`r`.`value`) AS `min_value` from `20100930_prices`.`site` `s` straight_join `20100930_prices`.`price` `r` straight_join `20100930_prices`.`product` `p` where ((`20100930_prices`.`r`.`site` = `20100930_prices`.`s`.`id`) and (`20100930_prices`.`p`.`id` = `20100930_prices`.`r`.`product`) and (`20100930_prices`.`p`.`parent` = 1) and (`20100930_prices`.`s`.`region` = 30)) group by `20100930_prices`.`p`.`id` order by min(`20100930_prices`.`r`.`value`) limit 10

This query completes in 30 ms, since the conditions (in their totality) are less selective.

Product, price, site

This query is best for sparse products and dense sites:

In this query, we will replace the JOIN against the site table with an IN predicate, since we don’t actually need any information from that table:

SELECT  p.*, MIN(r.value) AS min_value
FROM    product p
STRAIGHT_JOIN
        price r
ON      r.product = p.id
WHERE   p.parent = 100
        AND r.site IN
        (
        SELECT  id
        FROM    site
        WHERE   region = 1
        )
GROUP BY
        p.id
ORDER BY
        min_value
LIMIT 10

select `20100930_prices`.`p`.`id` AS `id`,`20100930_prices`.`p`.`parent` AS `parent`,`20100930_prices`.`p`.`name` AS `name`,min(`20100930_prices`.`r`.`value`) AS `min_value` from `20100930_prices`.`product` `p` straight_join `20100930_prices`.`price` `r` where ((`20100930_prices`.`r`.`product` = `20100930_prices`.`p`.`id`) and (`20100930_prices`.`p`.`parent` = 100) and <in_optimizer>(`20100930_prices`.`r`.`site`,<exists>(<primary_index_lookup>(<cache>(`20100930_prices`.`r`.`site`) in site on PRIMARY where ((`20100930_prices`.`site`.`region` = 1) and (<cache>(`20100930_prices`.`r`.`site`) = `20100930_prices`.`site`.`id`)))))) group by `20100930_prices`.`p`.`id` order by min(`20100930_prices`.`r`.`value`) limit 10

Again, this query is 30 ms.

Price, product/site

This query is best for dense products and dense sites. This is the most complex query which would require some explanation.

At the first sight, it may seem that price cannot be filtered on. However, it’s not true.

What we need is 10 lowest values from price satisfying certain conditions (right products and sites, and, additionally, products should be unique).

This means that we could make use of an index on price.value, scanning it until 10 records satisfying these conditions are returned. In this case, ORDER BY and LIMIT 10 would serve as filtering conditions: the scanning would cease as soon as the limit is reached.

And of course the more dense are the products and sites, the more is the probability for the conditions to be satisfied, the sooner the query completes.

Filtering on products and sites is easy, but to ensure that the products are unique, we will use the trick described in my previous article:

• 20 latest unique records

Basically, when filtering a price, we should not only make sure that it holds the correct product, but that it’s the first (lowest) price for a given product.

This can be easily done with an additional ORDER BY / LIMIT in a subquery.

Here’s the query:

SELECT  p.*, value AS min_value
FROM    price r
STRAIGHT_JOIN
        product p
ON      p.id = r.product
WHERE   p.parent = 1
        AND r.id =
        (
        SELECT  ri.id
        FROM    price ri
        WHERE   ri.product = p.id
                AND ri.site IN
                (
                SELECT  id
                FROM    site
                WHERE   region = 1
                )
        ORDER BY
                ri.product, ri.value, ri.id
        LIMIT 1
        )
ORDER BY
        r.value, r.id
LIMIT 10
        

Field or reference '20100930_prices.p.id' of SELECT #2 was resolved in SELECT #1
select `20100930_prices`.`p`.`id` AS `id`,`20100930_prices`.`p`.`parent` AS `parent`,`20100930_prices`.`p`.`name` AS `name`,`20100930_prices`.`r`.`value` AS `min_value` from `20100930_prices`.`price` `r` straight_join `20100930_prices`.`product` `p` where ((`20100930_prices`.`p`.`id` = `20100930_prices`.`r`.`product`) and (`20100930_prices`.`p`.`parent` = 1) and (`20100930_prices`.`r`.`id` = (select `20100930_prices`.`ri`.`id` from `20100930_prices`.`price` `ri` where ((`20100930_prices`.`ri`.`product` = `20100930_prices`.`p`.`id`) and <in_optimizer>(`20100930_prices`.`ri`.`site`,<exists>(<primary_index_lookup>(<cache>(`20100930_prices`.`ri`.`site`) in site on PRIMARY where ((`20100930_prices`.`site`.`region` = 1) and (<cache>(`20100930_prices`.`ri`.`site`) = `20100930_prices`.`site`.`id`)))))) order by `20100930_prices`.`ri`.`product`,`20100930_prices`.`ri`.`value`,`20100930_prices`.`ri`.`id` limit 1))) order by `20100930_prices`.`r`.`value`,`20100930_prices`.`r`.`id` limit 10

We see that this query completes in only 14 ms despite the fact that we used the most populated parent and region. As I said before, the LIMIT 10 served as another filtering condition.

Summary

Let’s run all possible queries for all possible distributions and record the query times in a table

P, S and R stand for product, site and price in the join order.

+Inf means that the query did not complete in a reasonable time and had to be terminated.

Conclusion

If limiting the results, MySQL allows filtering fact tables efficiently on higher level dimensions despite the fact these dimensions cannot be indexed. However, in each case the selectivity of the dimension level should be taken into account and the appropriate query should be used.

It should be noted that the queries involving a CROSS JOIN and index scan on the fact table perform intolerably poorly in the edge cases (too dense and too sparse dimensions, accordingly). On the other hand, the queries involving a join only differ in the join order which can be predicted by MySQL.

This means that when the dimension selectivity is unknown, a query using a plain join of all three tables (without forcing the join order) should be the query of choice:

SELECT  p.*, MIN(r.value) AS min_value
FROM    product p
CROSS JOIN
        site s
JOIN    price r
ON      r.product = p.id
        AND r.site = s.id
WHERE   p.parent = 1
        AND s.region = 1
GROUP BY
        p.id
ORDER BY
        min_value
LIMIT 10

, since MySQL can select a join order quite close to optimal and the query would complete in reasonable time.

However, when the distribution on both dimensions is known, the appropriate query from the table above can be chosen.

Hope that helps.

I’m always glad to answer the questions regarding database queries.

Ask me a question

I have a logfile which logs the insert/delete/updates from all kinds of tables.

I would like to get an overview of for example the last 20 people which records where updated, ordered by the last update (datetime DESC)

A common solution for such a task would be writing an aggregate query with ORDER BY and LIMIT:

SELECT  person, MAX(ts) AS last_update
FROM    logfile
GROUP BY
        person
ORDER BY
        last_update DESC
LIMIT 20

What’s bad in this solution? Performance, as usual.

Since last_update is an aggregate, it cannot be indexed. And ORDER BY on unindexed fields results in our good old friend, filesort.

Note that even in this case the indexes can be used and the full table scan can be avoided: if there is an index on (person, ts), MySQL will tend to use a loose index scan on this index, which can save this query if there are relatively few persons in the table. However, if there are many (which is what we can expect for a log table), loose index scan can even degrade performance and generally will be avoided by MySQL.

We should use another approach here. Let’s create a sample table and test this approach:

Table creation details

CREATE TABLE filler (
        id INT NOT NULL PRIMARY KEY AUTO_INCREMENT
) ENGINE=Memory;

CREATE TABLE logfile (
        id INT NOT NULL PRIMARY KEY,
        sparse INT NOT NULL,
        dense INT NOT NULL,
        ts DATETIME NOT NULL,
        stuffing VARCHAR(100) NOT NULL,
        KEY ix_logfile_ts_id (ts, id),
        KEY ix_logfile_sparse_ts_id (sparse, ts, id),
        KEY ix_logfile_dense_ts_id (dense, ts, id)
) ENGINE=InnoDB;
       
DELIMITER $$

CREATE PROCEDURE prc_filler(cnt INT)
BEGIN
        DECLARE _cnt INT;
        SET _cnt = 1;
        WHILE _cnt <= cnt DO
                INSERT
                INTO    filler
                SELECT  _cnt;
                SET _cnt = _cnt + 1;
        END WHILE;
END
$$

DELIMITER ;

START TRANSACTION;
CALL prc_filler(500000);
COMMIT;

INSERT
INTO    logfile
SELECT  id,
        CEILING(RAND(20100824) * 30),
        CEILING(RAND(20100824 << 1) * 30000),
        '2010-08-24' - INTERVAL RAND(20100824 << 2) * 10000000 SECOND,
        LPAD('', 100, '*')
FROM    filler;

This table has 1,000,000 records.

Instead of a single field, person, I created two different fields: sparse and dense. The first one has 30 distinct values, while the second one has 30,000. This will help us to see how data distribution affects performance of different queries.

Let’s run our original queries. We’ll adjust them a little to help MySQL to pick correct plans:

SELECT  *
FROM    (
        SELECT  sparse, MAX(ts) AS last_update
        FROM    logfile
        GROUP BY
                sparse
        ) q
ORDER BY
        last_update DESC
LIMIT 20;

View query results

select `q`.`sparse` AS `sparse`,`q`.`last_update` AS `last_update` from (select `20100824_latest`.`logfile`.`sparse` AS `sparse`,max(`20100824_latest`.`logfile`.`ts`) AS `last_update` from `20100824_latest`.`logfile` group by `20100824_latest`.`logfile`.`sparse`) `q` order by `q`.`last_update` desc limit 20

SELECT  *
FROM    (
        SELECT  dense, MAX(ts) AS last_update
        FROM    logfile
        GROUP BY
                dense
        ) q
ORDER BY
        last_update DESC
LIMIT 20;

View query results

select `q`.`dense` AS `dense`,`q`.`last_update` AS `last_update` from (select `20100824_latest`.`logfile`.`dense` AS `dense`,max(`20100824_latest`.`logfile`.`ts`) AS `last_update` from `20100824_latest`.`logfile` group by `20100824_latest`.`logfile`.`dense`) `q` order by `q`.`last_update` desc limit 20

We see that both queries use the same plan and return 20 records, but the first one is instant, while the second one runs for 500 ms. Both queries use filesort, but in second case it has to sort 30,000 records (compared to 30 in the first case).

In this case, it is better to use another approach.

With our original query, we take each person and see which record is latest for this person. But we can as well do it the other way round: take the records in descending order, one by one, and for each record see if it’s latest for this person. If it is, we should return it; if it’s not, this means that the record for this person has already been returned (remember, we take them in descending order).

It’s easy to see that 20 records returned this way will, first, belong to 20 different people, and, second, be the latest records of their respective persons. This is exactly what we need.

The records can easily be scanned in the descending order using the index on (ts, id). But how do we check if the record is the latest? It’s simple: we just take the last record for the given person from the index on (person, ts, id) and compare its id. It takes but a single index seek per record and is almost instant.

Here’s the query to do it:

SELECT  id, sparse, dense, ts
FROM    logfile lf
WHERE   id = 
        (
        SELECT  id
        FROM    logfile lfi
        WHERE   lfi.sparse = lf.sparse
        ORDER BY
                sparse DESC, ts DESC, id DESC
        LIMIT 1
        )
ORDER BY
        ts DESC, id DESC
LIMIT 20
        

Field or reference '20100824_latest.lf.sparse' of SELECT #2 was resolved in SELECT #1
select `20100824_latest`.`lf`.`id` AS `id`,`20100824_latest`.`lf`.`sparse` AS `sparse`,`20100824_latest`.`lf`.`dense` AS `dense`,`20100824_latest`.`lf`.`ts` AS `ts` from `20100824_latest`.`logfile` `lf` where (`20100824_latest`.`lf`.`id` = (select `20100824_latest`.`lfi`.`id` from `20100824_latest`.`logfile` `lfi` where (`20100824_latest`.`lfi`.`sparse` = `20100824_latest`.`lf`.`sparse`) order by `20100824_latest`.`lfi`.`sparse` desc,`20100824_latest`.`lfi`.`ts` desc,`20100824_latest`.`lfi`.`id` desc limit 1)) order by `20100824_latest`.`lf`.`ts` desc,`20100824_latest`.`lf`.`id` desc limit 20

As we can see, this query uses two different indexes. The first one on (ts, id) is used to scan all records according to the overall timeline; the second one, on (sparse, ts, id) is used to find the id of the latest entry for a person and check if it’s the same as the record selected from the general timeline.

The query is instant: 3 ms.

Let’s check the same query on a column with lots of values:

SELECT  id, sparse, dense, ts
FROM    logfile lf
WHERE   id = 
        (
        SELECT  id
        FROM    logfile lfi
        WHERE   lfi.dense = lf.dense
        ORDER BY
                dense DESC, ts DESC, id DESC
        LIMIT 1
        )
ORDER BY
        ts DESC, id DESC
LIMIT 20

Field or reference '20100824_latest.lf.dense' of SELECT #2 was resolved in SELECT #1
select `20100824_latest`.`lf`.`id` AS `id`,`20100824_latest`.`lf`.`sparse` AS `sparse`,`20100824_latest`.`lf`.`dense` AS `dense`,`20100824_latest`.`lf`.`ts` AS `ts` from `20100824_latest`.`logfile` `lf` where (`20100824_latest`.`lf`.`id` = (select `20100824_latest`.`lfi`.`id` from `20100824_latest`.`logfile` `lfi` where (`20100824_latest`.`lfi`.`dense` = `20100824_latest`.`lf`.`dense`) order by `20100824_latest`.`lfi`.`dense` desc,`20100824_latest`.`lfi`.`ts` desc,`20100824_latest`.`lfi`.`id` desc limit 1)) order by `20100824_latest`.`lf`.`ts` desc,`20100824_latest`.`lf`.`id` desc limit 20

We see that the query is instant again, despite the data distribution being completely different. This is because the query only skips the records which are not the latest of their persons, and the total number of the records to scan is defined by how many records do we browse before we encounter the 20th unique value in our scan. This value decreases exponentially as the number of distinct persons in the table grows, but with 99% probability it won’t exceed 100 records even for only 20 distinct persons in the table.

The only problem that can arise here is that the number of distinct persons in the table is less than the LIMIT we set. In this case, no new records after the limit is reached can be returned, and a full index scan (accompanied by an index seek once per record) will ultimately be performed.

To work around this, the following simple query can be run in advance:

SELECT  COUNT(*)
FROM    (
        SELECT  DISTINCT sparse
        FROM    logfile
        LIMIT 20
        ) q

View query details

select count(0) AS `COUNT(*)` from (select distinct `20100824_latest`.`logfile`.`sparse` AS `sparse` from `20100824_latest`.`logfile` limit 20) `q`

This query will return the actual number of distinct persons in the table if there are less than 20 (or 20 if these are more).

This query is instant even for the dense data:

SELECT  COUNT(*)
FROM    (
        SELECT  DISTINCT dense
        FROM    logfile
        LIMIT 20
        ) q

View query details

select count(0) AS `COUNT(*)` from (select distinct `20100824_latest`.`logfile`.`dense` AS `dense` from `20100824_latest`.`logfile` limit 20) `q`

This needs to be run as a separate query because MySQL does not allow using anything other than constants in the LIMIT clause. The result of this query should be substituted into the LIMIT clause on the client or in a dynamically composed query on the server.

Summary

To select a number of latest unique records from a table, one can use aggregate functions, however, this can decrease the query performance.

This can be done more efficiently by creating two different indexes on the table and checking the records taken from the general timeline against the end of the index on the person’s timeline.

To avoid performance degradation in marginal cases (when the total number of persons in the table is less than LIMIT), it is possible to make an additional check for the total number of distinct records and adjust the LIMIT clause if there are not enough records.

P. S. I decided to enable comments for the technical posts as well. You are welcome to comment.

Frode Underhill asks:

I have some applications that are logging to a MySQL database table.

The table is pretty standard on the form:

time BIGINT(20)	source TINYINT(4)	severity ENUM	text VARCHAR(255)

, where source identifies the system that generated the log entry.

There are very many entries in the table (>100 million), of which 99.9999% are debug or info messages.

I’m making an interface for browsing this log, which means I’ll be doing queries like

SELECT  *
FROM    log 
WHERE   source = 2
        AND severity IN (1,2) 
        AND time > 12345 
ORDER BY
        time ASC
LIMIT 30

, if I want to find debug or info log entries from a certain point in time, or

SELECT  *
FROM    log 
WHERE   source = 2
        AND severity IN (1,2) 
        AND time < 12345 
ORDER BY
        time DESC
LIMIT 30

for finding entries right before a certain time.

How would one go about indexing & querying such a table?

I thought I had it figured out (I pretty much just tried every different combination of columns in an index), but there’s always some set of parameters that results in a really slow query.

The problem is that you cannot use a single index both for filtering and ordering if you have a ranged condition (severity IN (1, 2) in this case).

Recently I wrote an article with a proposal to improve SQL optimizer to handle these conditions. If a range has low cardinality (this is, there are few values that con possibly satisfy the range), then the query could be improved by rewriting the range as a series of individual queries, each one using one of the values constituting the range in an equijoin:

• Things SQL needs: determining range cardinality

No optimizers can handle this condition automatically yet, so we’ll need to emulate it.

Since the severity field is defined as an enum with only 5 values possible, any range condition on this field can be satisfied by no more than 5 distinct values, thus making this table ideal for rewriting the query.

Let’s create a sample table:

Table creation details

CREATE TABLE filler (
        id INT NOT NULL PRIMARY KEY AUTO_INCREMENT
) ENGINE=Memory;

CREATE TABLE t_log (
        id INT NOT NULL,
        ts BIGINT NOT NULL,
        source TINYINT(4) NOT NULL,
        severity ENUM('DEBUG','INFO','WARNING','ERROR','FATAL') NOT NULL,
        tx VARCHAR(255)
) ENGINE=MyISAM;

DELIMITER $$

CREATE PROCEDURE prc_filler(cnt INT)
BEGIN
        DECLARE _cnt INT;
        SET _cnt = 1;
        WHILE _cnt <= cnt DO
                INSERT
                INTO    filler
                SELECT  _cnt;
                SET _cnt = _cnt + 1;
        END WHILE;
END
$$

DELIMITER ;

START TRANSACTION;
CALL prc_filler(3500);
COMMIT;

INSERT
INTO    t_log
SELECT  (f1.id - 1) * 3000 + f2.id,
        UNIX_TIMESTAMP('2010-06-29' - INTERVAL (f1.id - 1) * 3000 + f2.id SECOND),
        CEILING(RAND(20100629) * 10),
        5 - FLOOR(LOG10(CEILING(RAND(20100629 << 1) * 99999))),
        CONCAT('Message ', (f1.id - 1) * 3000 + f2.id)
FROM    filler f1
CROSS JOIN
        filler f2;

CREATE INDEX ix_log_source_ts ON t_log (source, ts);

CREATE INDEX ix_log_source_severity_ts ON t_log (source, severity, ts);

This MyISAM table has 12,250,000 records, with 10 random sources (distributed evenly) and 5 random severities (distributed logarithmically):

SELECT  severity, COUNT(*)
FROM    t_log
GROUP BY
        severity;

We also created two indexes (one on (source, ts), the other one on (source, severity, ts)).

Now, let’s try to run some queries as is:

SELECT  *
FROM    t_log
WHERE   source = 2
        AND severity IN (1, 2)
        AND ts <= 1277754000
ORDER BY
        source DESC, ts DESC
LIMIT 30

select `20100630_range`.`t_log`.`id` AS `id`,`20100630_range`.`t_log`.`ts` AS `ts`,`20100630_range`.`t_log`.`source` AS `source`,`20100630_range`.`t_log`.`severity` AS `severity`,`20100630_range`.`t_log`.`tx` AS `tx` from `20100630_range`.`t_log` where ((`20100630_range`.`t_log`.`source` = 2) and (`20100630_range`.`t_log`.`severity` in (1,2)) and (`20100630_range`.`t_log`.`ts` <= 1277754000)) order by `20100630_range`.`t_log`.`source` desc,`20100630_range`.`t_log`.`ts` desc limit 30

This is very fast. It uses the index which does not include severity: since the values 1 and 2 are very frequent, it’s much more efficient just to filter them out. The index preserves the order, that’s why there is no filesort in the plan.

SELECT  *
FROM    t_log
WHERE   source = 2
        AND severity IN (4, 5)
        AND ts <= 1277754000
ORDER BY
        source DESC, ts DESC
LIMIT 30

select `20100630_range`.`t_log`.`id` AS `id`,`20100630_range`.`t_log`.`ts` AS `ts`,`20100630_range`.`t_log`.`source` AS `source`,`20100630_range`.`t_log`.`severity` AS `severity`,`20100630_range`.`t_log`.`tx` AS `tx` from `20100630_range`.`t_log` where ((`20100630_range`.`t_log`.`source` = 2) and (`20100630_range`.`t_log`.`severity` in (4,5)) and (`20100630_range`.`t_log`.`ts` <= 1277754000)) order by `20100630_range`.`t_log`.`source` desc,`20100630_range`.`t_log`.`ts` desc limit 30

This is very fast too. The index which includes severity is used (along with the filesort of course, because the order cannot be preserved with multiple values of severity), but the total number of records evaluated is so small that the filesort is not much of a problem.

Now, let’s try to include 3 into the query above:

SELECT  *
FROM    t_log
WHERE   source = 2
        AND severity IN (3, 4)
        AND ts <= 1277754000
ORDER BY
        source DESC, ts DESC
LIMIT 30

select `20100630_range`.`t_log`.`id` AS `id`,`20100630_range`.`t_log`.`ts` AS `ts`,`20100630_range`.`t_log`.`source` AS `source`,`20100630_range`.`t_log`.`severity` AS `severity`,`20100630_range`.`t_log`.`tx` AS `tx` from `20100630_range`.`t_log` where ((`20100630_range`.`t_log`.`source` = 2) and (`20100630_range`.`t_log`.`severity` in (3,4)) and (`20100630_range`.`t_log`.`ts` <= 1277754000)) order by `20100630_range`.`t_log`.`source` desc,`20100630_range`.`t_log`.`ts` desc limit 30

Now, this runs for almost 250 ms. Why?

There are 110,557 records with severity = 'WARNING'. This is too many for a filesort but too few for using where (filtering the records with the index that preserves the order). There will be too many records that will need to be skipped.

To work around this, we could combine the queries using UNION ALL. Since the original query uses ORDER BY and LIMIT, we may put them into two separate queries (which will yield 60 records) and finally apply it to the end resultset (to get the 30 records that are guaranteed to be contained among these 60):

SELECT  *
FROM    (
        SELECT  *
        FROM    t_log
        WHERE   source = 2
                AND severity = 3
                AND ts <= 1277754000
        ORDER BY
                source DESC, ts DESC
        LIMIT 30
        ) q
UNION ALL
SELECT  *
FROM    (
        SELECT  *
        FROM    t_log
        WHERE   source = 2
                AND severity = 4
                AND ts <= 1277754000
        ORDER BY
                source DESC, ts DESC
        LIMIT 30
        ) q
ORDER BY
        ts DESC
LIMIT 30

select `q`.`id` AS `id`,`q`.`ts` AS `ts`,`q`.`source` AS `source`,`q`.`severity` AS `severity`,`q`.`tx` AS `tx` from (select `20100630_range`.`t_log`.`id` AS `id`,`20100630_range`.`t_log`.`ts` AS `ts`,`20100630_range`.`t_log`.`source` AS `source`,`20100630_range`.`t_log`.`severity` AS `severity`,`20100630_range`.`t_log`.`tx` AS `tx` from `20100630_range`.`t_log` where ((`20100630_range`.`t_log`.`source` = 2) and (`20100630_range`.`t_log`.`severity` = 3) and (`20100630_range`.`t_log`.`ts` <= 1277754000)) order by `20100630_range`.`t_log`.`source` desc,`20100630_range`.`t_log`.`ts` desc limit 30) `q` union all select `q`.`id` AS `id`,`q`.`ts` AS `ts`,`q`.`source` AS `source`,`q`.`severity` AS `severity`,`q`.`tx` AS `tx` from (select `20100630_range`.`t_log`.`id` AS `id`,`20100630_range`.`t_log`.`ts` AS `ts`,`20100630_range`.`t_log`.`source` AS `source`,`20100630_range`.`t_log`.`severity` AS `severity`,`20100630_range`.`t_log`.`tx` AS `tx` from `20100630_range`.`t_log` where ((`20100630_range`.`t_log`.`source` = 2) and (`20100630_range`.`t_log`.`severity` = 4) and (`20100630_range`.`t_log`.`ts` <= 1277754000)) order by `20100630_range`.`t_log`.`source` desc,`20100630_range`.`t_log`.`ts` desc limit 30) `q` order by `ts` desc limit 30

This is much faster.

However, this solution requires composing the query dynamically, depending on the number of the severities in the condition. Is it possible to make this all in one static query that will accept the parameters in the IN list?

We can do it by using the applying the solution using to retrieve greatest-n-per-group in MySQL.

To do this, we will just select the 30th timestamp of each severity and find all records with the higher timestamps.

This can be done using a join:

SELECT  *
FROM    (
        SELECT  l.*
        FROM    (
                SELECT  source,
                        severity,
                        (
                        SELECT  ts
                        FROM    t_log li
                        WHERE   li.source = ss.source
                                AND li.severity = ss.severity
                                AND ts <= 1277754000
                        ORDER BY
                                li.source DESC, li.severity DESC, li.ts DESC
                        LIMIT 29, 1
                        ) AS mts
                FROM    (
                        SELECT  DISTINCT source, severity
                        FROM    t_log
                        WHERE   source = 2
                                AND severity IN (3, 4)
                        ) ss
                ) s
        JOIN    t_log l
        ON      l.source >= s.source
                AND l.source <= s.source
                AND l.severity = s.severity
                AND l.ts >= s.mts
                AND l.ts <= 1277754000
        ) q
ORDER BY
        ts DESC
LIMIT 30;

Field or reference 'ss.source' of SELECT #4 was resolved in SELECT #3
Field or reference 'ss.severity' of SELECT #4 was resolved in SELECT #3
select `q`.`id` AS `id`,`q`.`ts` AS `ts`,`q`.`source` AS `source`,`q`.`severity` AS `severity`,`q`.`tx` AS `tx` from (select `20100630_range`.`l`.`id` AS `id`,`20100630_range`.`l`.`ts` AS `ts`,`20100630_range`.`l`.`source` AS `source`,`20100630_range`.`l`.`severity` AS `severity`,`20100630_range`.`l`.`tx` AS `tx` from (select `ss`.`source` AS `source`,`ss`.`severity` AS `severity`,(select `20100630_range`.`li`.`ts` from `20100630_range`.`t_log` `li` where ((`20100630_range`.`li`.`source` = `ss`.`source`) and (`20100630_range`.`li`.`severity` = `ss`.`severity`) and (`20100630_range`.`li`.`ts` <= 1277754000)) order by `20100630_range`.`li`.`source` desc,`20100630_range`.`li`.`severity` desc,`20100630_range`.`li`.`ts` desc limit 29,1) AS `mts` from (select distinct `20100630_range`.`t_log`.`source` AS `source`,`20100630_range`.`t_log`.`severity` AS `severity` from `20100630_range`.`t_log` where ((`20100630_range`.`t_log`.`source` = 2) and (`20100630_range`.`t_log`.`severity` in (3,4)))) `ss`) `s` join `20100630_range`.`t_log` `l` where ((`20100630_range`.`l`.`severity` = `s`.`severity`) and (`20100630_range`.`l`.`source` >= `s`.`source`) and (`20100630_range`.`l`.`source` <= `s`.`source`) and (`20100630_range`.`l`.`ts` >= `s`.`mts`) and (`20100630_range`.`l`.`ts` <= 1277754000))) `q` order by `q`.`ts` desc limit 30

All possible values of source and severity are selected using a loose scan (which is instant since there are few of them). Each pair of values is then used as a join condition. A single index range satisfies each pair of values, so each join iteration uses an index efficiently (actually, the access path is reevaluated for each iteration as shown by Range checked for each record (index map: 0x3).

The total number of records that would be returned by this query be there no LIMIT is 60 or maybe even more (in case of ties on ts). However, we don’t need to resolve the ties in the subqueries, since the final ORDER BY / LIMIT does this for us.

The query completes in 4 ms which is instant. More than that, it does not need to be rewritten to handle different combinations of values: they could be provided in a single IN clause.

Hope that helps.

I’m always glad to answer the questions regarding database queries.

Ask me a question