I have a logfile which logs the insert/delete/updates from all kinds of tables.

I would like to get an overview of for example the last 20 people which records where updated, ordered by the last update (datetime DESC)

A common solution for such a task would be writing an aggregate query with ORDER BY and LIMIT:

SELECT  person, MAX(ts) AS last_update
FROM    logfile
GROUP BY
        person
ORDER BY
        last_update DESC
LIMIT 20

What’s bad in this solution? Performance, as usual.

Since last_update is an aggregate, it cannot be indexed. And ORDER BY on unindexed fields results in our good old friend, filesort.

Note that even in this case the indexes can be used and the full table scan can be avoided: if there is an index on (person, ts), MySQL will tend to use a loose index scan on this index, which can save this query if there are relatively few persons in the table. However, if there are many (which is what we can expect for a log table), loose index scan can even degrade performance and generally will be avoided by MySQL.

We should use another approach here. Let’s create a sample table and test this approach:

Table creation details

CREATE TABLE filler (
        id INT NOT NULL PRIMARY KEY AUTO_INCREMENT
) ENGINE=Memory;

CREATE TABLE logfile (
        id INT NOT NULL PRIMARY KEY,
        sparse INT NOT NULL,
        dense INT NOT NULL,
        ts DATETIME NOT NULL,
        stuffing VARCHAR(100) NOT NULL,
        KEY ix_logfile_ts_id (ts, id),
        KEY ix_logfile_sparse_ts_id (sparse, ts, id),
        KEY ix_logfile_dense_ts_id (dense, ts, id)
) ENGINE=InnoDB;

DELIMITER $$

CREATE PROCEDURE prc_filler(cnt INT)
BEGIN
        DECLARE _cnt INT;
        SET _cnt = 1;
        WHILE _cnt <= cnt DO
                INSERT
                INTO    filler
                SELECT  _cnt;
                SET _cnt = _cnt + 1;
        END WHILE;
END
$$

DELIMITER ;

START TRANSACTION;
CALL prc_filler(500000);
COMMIT;

INSERT
INTO    logfile
SELECT  id,
        CEILING(RAND(20100824) * 30),
        CEILING(RAND(20100824 << 1) * 30000),
        '2010-08-24' - INTERVAL RAND(20100824 << 2) * 10000000 SECOND,
        LPAD('', 100, '*')
FROM    filler;

This table has 1,000,000 records.

Instead of a single field, person, I created two different fields: sparse and dense. The first one has 30 distinct values, while the second one has 30,000. This will help us to see how data distribution affects performance of different queries.

Let’s run our original queries. We’ll adjust them a little to help MySQL to pick correct plans:

SELECT  *
FROM    (
        SELECT  sparse, MAX(ts) AS last_update
        FROM    logfile
        GROUP BY
                sparse
        ) q
ORDER BY
        last_update DESC
LIMIT 20;

View query results

select `q`.`sparse` AS `sparse`,`q`.`last_update` AS `last_update` from (select `20100824_latest`.`logfile`.`sparse` AS `sparse`,max(`20100824_latest`.`logfile`.`ts`) AS `last_update` from `20100824_latest`.`logfile` group by `20100824_latest`.`logfile`.`sparse`) `q` order by `q`.`last_update` desc limit 20

SELECT  *
FROM    (
        SELECT  dense, MAX(ts) AS last_update
        FROM    logfile
        GROUP BY
                dense
        ) q
ORDER BY
        last_update DESC
LIMIT 20;

View query results

select `q`.`dense` AS `dense`,`q`.`last_update` AS `last_update` from (select `20100824_latest`.`logfile`.`dense` AS `dense`,max(`20100824_latest`.`logfile`.`ts`) AS `last_update` from `20100824_latest`.`logfile` group by `20100824_latest`.`logfile`.`dense`) `q` order by `q`.`last_update` desc limit 20

We see that both queries use the same plan and return 20 records, but the first one is instant, while the second one runs for 500 ms. Both queries use filesort, but in second case it has to sort 30,000 records (compared to 30 in the first case).

In this case, it is better to use another approach.

With our original query, we take each person and see which record is latest for this person. But we can as well do it the other way round: take the records in descending order, one by one, and for each record see if it’s latest for this person. If it is, we should return it; if it’s not, this means that the record for this person has already been returned (remember, we take them in descending order).

It’s easy to see that 20 records returned this way will, first, belong to 20 different people, and, second, be the latest records of their respective persons. This is exactly what we need.

The records can easily be scanned in the descending order using the index on (ts, id). But how do we check if the record is the latest? It’s simple: we just take the last record for the given person from the index on (person, ts, id) and compare its id. It takes but a single index seek per record and is almost instant.

Here’s the query to do it:

SELECT  id, sparse, dense, ts
FROM    logfile lf
WHERE   id =
        (
        SELECT  id
        FROM    logfile lfi
        WHERE   lfi.sparse = lf.sparse
        ORDER BY
                sparse DESC, ts DESC, id DESC
        LIMIT 1
        )
ORDER BY
        ts DESC, id DESC
LIMIT 20

Field or reference '20100824_latest.lf.sparse' of SELECT #2 was resolved in SELECT #1
select `20100824_latest`.`lf`.`id` AS `id`,`20100824_latest`.`lf`.`sparse` AS `sparse`,`20100824_latest`.`lf`.`dense` AS `dense`,`20100824_latest`.`lf`.`ts` AS `ts` from `20100824_latest`.`logfile` `lf` where (`20100824_latest`.`lf`.`id` = (select `20100824_latest`.`lfi`.`id` from `20100824_latest`.`logfile` `lfi` where (`20100824_latest`.`lfi`.`sparse` = `20100824_latest`.`lf`.`sparse`) order by `20100824_latest`.`lfi`.`sparse` desc,`20100824_latest`.`lfi`.`ts` desc,`20100824_latest`.`lfi`.`id` desc limit 1)) order by `20100824_latest`.`lf`.`ts` desc,`20100824_latest`.`lf`.`id` desc limit 20

As we can see, this query uses two different indexes. The first one on (ts, id) is used to scan all records according to the overall timeline; the second one, on (sparse, ts, id) is used to find the id of the latest entry for a person and check if it’s the same as the record selected from the general timeline.

The query is instant: 3 ms.

Let’s check the same query on a column with lots of values:

SELECT  id, sparse, dense, ts
FROM    logfile lf
WHERE   id =
        (
        SELECT  id
        FROM    logfile lfi
        WHERE   lfi.dense = lf.dense
        ORDER BY
                dense DESC, ts DESC, id DESC
        LIMIT 1
        )
ORDER BY
        ts DESC, id DESC
LIMIT 20

Field or reference '20100824_latest.lf.dense' of SELECT #2 was resolved in SELECT #1
select `20100824_latest`.`lf`.`id` AS `id`,`20100824_latest`.`lf`.`sparse` AS `sparse`,`20100824_latest`.`lf`.`dense` AS `dense`,`20100824_latest`.`lf`.`ts` AS `ts` from `20100824_latest`.`logfile` `lf` where (`20100824_latest`.`lf`.`id` = (select `20100824_latest`.`lfi`.`id` from `20100824_latest`.`logfile` `lfi` where (`20100824_latest`.`lfi`.`dense` = `20100824_latest`.`lf`.`dense`) order by `20100824_latest`.`lfi`.`dense` desc,`20100824_latest`.`lfi`.`ts` desc,`20100824_latest`.`lfi`.`id` desc limit 1)) order by `20100824_latest`.`lf`.`ts` desc,`20100824_latest`.`lf`.`id` desc limit 20

We see that the query is instant again, despite the data distribution being completely different. This is because the query only skips the records which are not the latest of their persons, and the total number of the records to scan is defined by how many records do we browse before we encounter the 20th unique value in our scan. This value decreases exponentially as the number of distinct persons in the table grows, but with 99% probability it won’t exceed 100 records even for only 20 distinct persons in the table.

The only problem that can arise here is that the number of distinct persons in the table is less than the LIMIT we set. In this case, no new records after the limit is reached can be returned, and a full index scan (accompanied by an index seek once per record) will ultimately be performed.

To work around this, the following simple query can be run in advance:

SELECT  COUNT(*)
FROM    (
        SELECT  DISTINCT sparse
        FROM    logfile
        LIMIT 20
        ) q

View query details

select count(0) AS `COUNT(*)` from (select distinct `20100824_latest`.`logfile`.`sparse` AS `sparse` from `20100824_latest`.`logfile` limit 20) `q`

This query will return the actual number of distinct persons in the table if there are less than 20 (or 20 if these are more).

This query is instant even for the dense data:

SELECT  COUNT(*)
FROM    (
        SELECT  DISTINCT dense
        FROM    logfile
        LIMIT 20
        ) q

View query details

select count(0) AS `COUNT(*)` from (select distinct `20100824_latest`.`logfile`.`dense` AS `dense` from `20100824_latest`.`logfile` limit 20) `q`

This needs to be run as a separate query because MySQL does not allow using anything other than constants in the LIMIT clause. The result of this query should be substituted into the LIMIT clause on the client or in a dynamically composed query on the server.

Summary

To select a number of latest unique records from a table, one can use aggregate functions, however, this can decrease the query performance.

This can be done more efficiently by creating two different indexes on the table and checking the records taken from the general timeline against the end of the index on the person’s timeline.

To avoid performance degradation in marginal cases (when the total number of persons in the table is less than LIMIT), it is possible to make an additional check for the total number of distinct records and adjust the LIMIT clause if there are not enough records.

P. S. I decided to enable comments for the technical posts as well. You are welcome to comment.

Pawel, Mark, Kate, Felix, Andrew, Ludwig, Santos, Jeremy, John, another John, Anjan and Dave — I do remember about you, guys!

Will try to resume the normal blogging schedule the next week.

Stay subscribed!

Frode Underhill asks:

I have some applications that are logging to a MySQL database table.

The table is pretty standard on the form:

time BIGINT(20)	source TINYINT(4)	severity ENUM	text VARCHAR(255)

, where source identifies the system that generated the log entry.

There are very many entries in the table (>100 million), of which 99.9999% are debug or info messages.

I’m making an interface for browsing this log, which means I’ll be doing queries like

SELECT  *
FROM    log
WHERE   source = 2
        AND severity IN (1,2)
        AND time > 12345
ORDER BY
        time ASC
LIMIT 30

, if I want to find debug or info log entries from a certain point in time, or

SELECT  *
FROM    log
WHERE   source = 2
        AND severity IN (1,2)
        AND time < 12345
ORDER BY
        time DESC
LIMIT 30

for finding entries right before a certain time.

How would one go about indexing & querying such a table?

I thought I had it figured out (I pretty much just tried every different combination of columns in an index), but there’s always some set of parameters that results in a really slow query.

The problem is that you cannot use a single index both for filtering and ordering if you have a ranged condition (severity IN (1, 2) in this case).

Recently I wrote an article with a proposal to improve SQL optimizer to handle these conditions. If a range has low cardinality (this is, there are few values that con possibly satisfy the range), then the query could be improved by rewriting the range as a series of individual queries, each one using one of the values constituting the range in an equijoin:

• Things SQL needs: determining range cardinality

No optimizers can handle this condition automatically yet, so we’ll need to emulate it.

Since the severity field is defined as an enum with only 5 values possible, any range condition on this field can be satisfied by no more than 5 distinct values, thus making this table ideal for rewriting the query.

Let’s create a sample table:

Table creation details

CREATE TABLE filler (
        id INT NOT NULL PRIMARY KEY AUTO_INCREMENT
) ENGINE=Memory;

CREATE TABLE t_log (
        id INT NOT NULL,
        ts BIGINT NOT NULL,
        source TINYINT(4) NOT NULL,
        severity ENUM('DEBUG','INFO','WARNING','ERROR','FATAL') NOT NULL,
        tx VARCHAR(255)
) ENGINE=MyISAM;

DELIMITER $$

CREATE PROCEDURE prc_filler(cnt INT)
BEGIN
        DECLARE _cnt INT;
        SET _cnt = 1;
        WHILE _cnt <= cnt DO
                INSERT
                INTO    filler
                SELECT  _cnt;
                SET _cnt = _cnt + 1;
        END WHILE;
END
$$

DELIMITER ;

START TRANSACTION;
CALL prc_filler(3500);
COMMIT;

INSERT
INTO    t_log
SELECT  (f1.id - 1) * 3000 + f2.id,
        UNIX_TIMESTAMP('2010-06-29' - INTERVAL (f1.id - 1) * 3000 + f2.id SECOND),
        CEILING(RAND(20100629) * 10),
        5 - FLOOR(LOG10(CEILING(RAND(20100629 << 1) * 99999))),
        CONCAT('Message ', (f1.id - 1) * 3000 + f2.id)
FROM    filler f1
CROSS JOIN
        filler f2;

CREATE INDEX ix_log_source_ts ON t_log (source, ts);

CREATE INDEX ix_log_source_severity_ts ON t_log (source, severity, ts);

This MyISAM table has 12,250,000 records, with 10 random sources (distributed evenly) and 5 random severities (distributed logarithmically):

SELECT  severity, COUNT(*)
FROM    t_log
GROUP BY
        severity;

We also created two indexes (one on (source, ts), the other one on (source, severity, ts)).

Now, let’s try to run some queries as is:

SELECT  *
FROM    t_log
WHERE   source = 2
        AND severity IN (1, 2)
        AND ts <= 1277754000
ORDER BY
        source DESC, ts DESC
LIMIT 30

select `20100630_range`.`t_log`.`id` AS `id`,`20100630_range`.`t_log`.`ts` AS `ts`,`20100630_range`.`t_log`.`source` AS `source`,`20100630_range`.`t_log`.`severity` AS `severity`,`20100630_range`.`t_log`.`tx` AS `tx` from `20100630_range`.`t_log` where ((`20100630_range`.`t_log`.`source` = 2) and (`20100630_range`.`t_log`.`severity` in (1,2)) and (`20100630_range`.`t_log`.`ts` <= 1277754000)) order by `20100630_range`.`t_log`.`source` desc,`20100630_range`.`t_log`.`ts` desc limit 30

This is very fast. It uses the index which does not include severity: since the values 1 and 2 are very frequent, it’s much more efficient just to filter them out. The index preserves the order, that’s why there is no filesort in the plan.

SELECT  *
FROM    t_log
WHERE   source = 2
        AND severity IN (4, 5)
        AND ts <= 1277754000
ORDER BY
        source DESC, ts DESC
LIMIT 30

select `20100630_range`.`t_log`.`id` AS `id`,`20100630_range`.`t_log`.`ts` AS `ts`,`20100630_range`.`t_log`.`source` AS `source`,`20100630_range`.`t_log`.`severity` AS `severity`,`20100630_range`.`t_log`.`tx` AS `tx` from `20100630_range`.`t_log` where ((`20100630_range`.`t_log`.`source` = 2) and (`20100630_range`.`t_log`.`severity` in (4,5)) and (`20100630_range`.`t_log`.`ts` <= 1277754000)) order by `20100630_range`.`t_log`.`source` desc,`20100630_range`.`t_log`.`ts` desc limit 30

This is very fast too. The index which includes severity is used (along with the filesort of course, because the order cannot be preserved with multiple values of severity), but the total number of records evaluated is so small that the filesort is not much of a problem.

Now, let’s try to include 3 into the query above:

SELECT  *
FROM    t_log
WHERE   source = 2
        AND severity IN (3, 4)
        AND ts <= 1277754000
ORDER BY
        source DESC, ts DESC
LIMIT 30

select `20100630_range`.`t_log`.`id` AS `id`,`20100630_range`.`t_log`.`ts` AS `ts`,`20100630_range`.`t_log`.`source` AS `source`,`20100630_range`.`t_log`.`severity` AS `severity`,`20100630_range`.`t_log`.`tx` AS `tx` from `20100630_range`.`t_log` where ((`20100630_range`.`t_log`.`source` = 2) and (`20100630_range`.`t_log`.`severity` in (3,4)) and (`20100630_range`.`t_log`.`ts` <= 1277754000)) order by `20100630_range`.`t_log`.`source` desc,`20100630_range`.`t_log`.`ts` desc limit 30

Now, this runs for almost 250 ms. Why?

There are 110,557 records with severity = 'WARNING'. This is too many for a filesort but too few for using where (filtering the records with the index that preserves the order). There will be too many records that will need to be skipped.

To work around this, we could combine the queries using UNION ALL. Since the original query uses ORDER BY and LIMIT, we may put them into two separate queries (which will yield 60 records) and finally apply it to the end resultset (to get the 30 records that are guaranteed to be contained among these 60):

SELECT  *
FROM    (
        SELECT  *
        FROM    t_log
        WHERE   source = 2
                AND severity = 3
                AND ts <= 1277754000
        ORDER BY
                source DESC, ts DESC
        LIMIT 30
        ) q
UNION ALL
SELECT  *
FROM    (
        SELECT  *
        FROM    t_log
        WHERE   source = 2
                AND severity = 4
                AND ts <= 1277754000
        ORDER BY
                source DESC, ts DESC
        LIMIT 30
        ) q
ORDER BY
        ts DESC
LIMIT 30

select `q`.`id` AS `id`,`q`.`ts` AS `ts`,`q`.`source` AS `source`,`q`.`severity` AS `severity`,`q`.`tx` AS `tx` from (select `20100630_range`.`t_log`.`id` AS `id`,`20100630_range`.`t_log`.`ts` AS `ts`,`20100630_range`.`t_log`.`source` AS `source`,`20100630_range`.`t_log`.`severity` AS `severity`,`20100630_range`.`t_log`.`tx` AS `tx` from `20100630_range`.`t_log` where ((`20100630_range`.`t_log`.`source` = 2) and (`20100630_range`.`t_log`.`severity` = 3) and (`20100630_range`.`t_log`.`ts` <= 1277754000)) order by `20100630_range`.`t_log`.`source` desc,`20100630_range`.`t_log`.`ts` desc limit 30) `q` union all select `q`.`id` AS `id`,`q`.`ts` AS `ts`,`q`.`source` AS `source`,`q`.`severity` AS `severity`,`q`.`tx` AS `tx` from (select `20100630_range`.`t_log`.`id` AS `id`,`20100630_range`.`t_log`.`ts` AS `ts`,`20100630_range`.`t_log`.`source` AS `source`,`20100630_range`.`t_log`.`severity` AS `severity`,`20100630_range`.`t_log`.`tx` AS `tx` from `20100630_range`.`t_log` where ((`20100630_range`.`t_log`.`source` = 2) and (`20100630_range`.`t_log`.`severity` = 4) and (`20100630_range`.`t_log`.`ts` <= 1277754000)) order by `20100630_range`.`t_log`.`source` desc,`20100630_range`.`t_log`.`ts` desc limit 30) `q` order by `ts` desc limit 30

This is much faster.

However, this solution requires composing the query dynamically, depending on the number of the severities in the condition. Is it possible to make this all in one static query that will accept the parameters in the IN list?

We can do it by using the applying the solution using to retrieve greatest-n-per-group in MySQL.

To do this, we will just select the 30th timestamp of each severity and find all records with the higher timestamps.

This can be done using a join:

SELECT  *
FROM    (
        SELECT  l.*
        FROM    (
                SELECT  source,
                        severity,
                        (
                        SELECT  ts
                        FROM    t_log li
                        WHERE   li.source = ss.source
                                AND li.severity = ss.severity
                                AND ts <= 1277754000
                        ORDER BY
                                li.source DESC, li.severity DESC, li.ts DESC
                        LIMIT 29, 1
                        ) AS mts
                FROM    (
                        SELECT  DISTINCT source, severity
                        FROM    t_log
                        WHERE   source = 2
                                AND severity IN (3, 4)
                        ) ss
                ) s
        JOIN    t_log l
        ON      l.source >= s.source
                AND l.source <= s.source
                AND l.severity = s.severity
                AND l.ts >= s.mts
                AND l.ts <= 1277754000
        ) q
ORDER BY
        ts DESC
LIMIT 30;

Field or reference 'ss.source' of SELECT #4 was resolved in SELECT #3
Field or reference 'ss.severity' of SELECT #4 was resolved in SELECT #3
select `q`.`id` AS `id`,`q`.`ts` AS `ts`,`q`.`source` AS `source`,`q`.`severity` AS `severity`,`q`.`tx` AS `tx` from (select `20100630_range`.`l`.`id` AS `id`,`20100630_range`.`l`.`ts` AS `ts`,`20100630_range`.`l`.`source` AS `source`,`20100630_range`.`l`.`severity` AS `severity`,`20100630_range`.`l`.`tx` AS `tx` from (select `ss`.`source` AS `source`,`ss`.`severity` AS `severity`,(select `20100630_range`.`li`.`ts` from `20100630_range`.`t_log` `li` where ((`20100630_range`.`li`.`source` = `ss`.`source`) and (`20100630_range`.`li`.`severity` = `ss`.`severity`) and (`20100630_range`.`li`.`ts` <= 1277754000)) order by `20100630_range`.`li`.`source` desc,`20100630_range`.`li`.`severity` desc,`20100630_range`.`li`.`ts` desc limit 29,1) AS `mts` from (select distinct `20100630_range`.`t_log`.`source` AS `source`,`20100630_range`.`t_log`.`severity` AS `severity` from `20100630_range`.`t_log` where ((`20100630_range`.`t_log`.`source` = 2) and (`20100630_range`.`t_log`.`severity` in (3,4)))) `ss`) `s` join `20100630_range`.`t_log` `l` where ((`20100630_range`.`l`.`severity` = `s`.`severity`) and (`20100630_range`.`l`.`source` >= `s`.`source`) and (`20100630_range`.`l`.`source` <= `s`.`source`) and (`20100630_range`.`l`.`ts` >= `s`.`mts`) and (`20100630_range`.`l`.`ts` <= 1277754000))) `q` order by `q`.`ts` desc limit 30

All possible values of source and severity are selected using a loose scan (which is instant since there are few of them). Each pair of values is then used as a join condition. A single index range satisfies each pair of values, so each join iteration uses an index efficiently (actually, the access path is reevaluated for each iteration as shown by Range checked for each record (index map: 0x3).

The total number of records that would be returned by this query be there no LIMIT is 60 or maybe even more (in case of ties on ts). However, we don’t need to resolve the ties in the subqueries, since the final ORDER BY / LIMIT does this for us.

The query completes in 4 ms which is instant. More than that, it does not need to be rewritten to handle different combinations of values: they could be provided in a single IN clause.

Hope that helps.

I’m always glad to answer the questions regarding database queries.

Ask me a question

Now, to the questions.

Mahen asks:

Create a table called Group:

Group

id	prodname
1	X
1	Y
1	Z
2	A
2	B
2	C

The resultset should look like this:

id	prodname
1	X,Y,Z
2	A,B,C

Can you please help me to solve the above problem using a recursive CTE?

This is out good old friend, GROUP_CONCAT. It’s an aggregate function that returns all strings within a group, concatenated. It’s somewhat different from the other aggregate functions, because, first, dealing with the concatenated string can be quite a tedious task for the groups with lots of records (large strings tend to overflow), and, second, the result depends on the order of the arguments (which is normally not the case for the aggregate functions). It’s not a part of a standard SQL and as for now is implemented only by MySQL with some extra vendor-specific keywords (like ORDER BY within the argument list).

This functionality, however, is often asked for and I have written some articles about implementing this in PostgreSQL and Oracle.

Now, let’s see how to do it in SQL Server.

Usually, SQL Server‘s FOR XML clause is exploited to concatenate the strings. To do this, we obtain a list of group identifiers and for each group, retrieve all it’s product names with a subquery appended with FOR XML PATH(''). This makes a single XML column out of the recordset:

WITH    q (id, prodname) AS
        (
        SELECT  1, 'X'
        UNION ALL
        SELECT  1, 'Y'
        UNION ALL
        SELECT  1, 'Z'
        UNION ALL
        SELECT  2, 'A'
        UNION ALL
        SELECT  2, 'B'
        UNION ALL
        SELECT  2, 'C'
        )
SELECT  *
FROM    (
        SELECT  DISTINCT id
        FROM    q
        ) qo
CROSS APPLY
        (
        SELECT  CASE ROW_NUMBER() OVER(ORDER BY prodname) WHEN 1 THEN '' ELSE ', ' END + qi.prodname
        FROM    q qi
        WHERE   qi.id = qo.id
        ORDER BY
                prodname
        FOR XML PATH ('')
        ) qi(r)

This solution works, but converting to and from XML is not the best way to deal with the strings: things like ampersands, angle brackets, line feeds etc. get mangled and require some additional effort to cope with.

However, this functionality can really be implemented using a recursive CTE.

To do this, we need to do the following:

1. Assign a group-wise row number and a group-wise count to each record (in required order)
2. Select the first record (that with the row number 1) from each group in the anchor part of the CTE
3. Recursively append the next record to the previous record. The next record can be obtained by joining on rn = rn + 1
4. Finally, select the only last record from each group (whose row number is equal to the group-wise count). It will contain the accumulated string.

Here’s how we do it:

WITH    q (id, prodname) AS
        (
        SELECT  1, 'X'
        UNION ALL
        SELECT  1, 'Y'
        UNION ALL
        SELECT  1, 'Z'
        UNION ALL
        SELECT  2, 'A'
        UNION ALL
        SELECT  2, 'B'
        UNION ALL
        SELECT  2, 'C'
        ),
        qs(id, prodname, rn, cnt) AS
        (
        SELECT  id, prodname,
                ROW_NUMBER() OVER (PARTITION BY id ORDER BY prodname),
                COUNT(*) OVER (PARTITION BY id)
        FROM    q
        ),
        t (id, prodname, gc, rn, cnt) AS
        (
        SELECT  id, prodname,
                CAST(prodname AS NVARCHAR(MAX)), rn, cnt
        FROM    qs
        WHERE   rn = 1
        UNION ALL
        SELECT  qs.id, qs.prodname,
                CAST(t.gc + ', ' + qs.prodname AS NVARCHAR(MAX)),
                qs.rn, qs.cnt
        FROM    t
        JOIN    qs
        ON      qs.id = t.id
                AND qs.rn = t.rn + 1
        )
SELECT  id, gc
FROM    t
WHERE   rn = cnt
OPTION (MAXRECURSION 0)

As we can see, this only deals with native NVARCHAR and is free from XML conversions.

Hope that helps.

I’m always glad to answer the questions regarding database queries.

Ask me a question

Just a quick reminder: an anti-join is an operation that returns all records from one table which share a value of a certain column with no records from another table.

In SQL, there are at least three methods to implement it:

LEFT JOIN / IS NULL

SELECT  o.*
FROM    outer o
LEFT JOIN
        inner i
ON      i.value = o.value
WHERE   i.value IS NULL

NOT IN

SELECT  o.*
FROM    outer o
WHERE   o.value NOT IN
        (
        SELECT  value
        FROM    inner
        )

NOT EXISTS

SELECT  o.*
FROM    outer o
WHERE   NOT EXISTS
        (
        SELECT  NULL
        FROM    inner i
        WHERE   i.value = o.value
        )

When inner.value is marked as NOT NULL, all these queries are semantically equivalent and with proper indexing have similarly optimized execution plans in MySQL.

Now, what if inner.value is not nullable and does contain some NULL values?

Let’s create some sample tables:

Table creation details

CREATE TABLE filler (
        id INT NOT NULL PRIMARY KEY AUTO_INCREMENT
) ENGINE=MyISAM;

CREATE TABLE t_inner (
        id INT NOT NULL PRIMARY KEY,
        val INT,
        stuffing VARCHAR(200) NOT NULL,
        KEY ix_inner_val (val)
) ENGINE=MyISAM DEFAULT CHARSET=utf8;

CREATE TABLE t_outer (
        id INT NOT NULL PRIMARY KEY,
        val INT,
        stuffing VARCHAR(200) NOT NULL,
        KEY ix_outer_val (val)
) ENGINE=MyISAM DEFAULT CHARSET=utf8;

DELIMITER $$

CREATE PROCEDURE prc_filler(cnt INT)
BEGIN
        DECLARE _cnt INT;
        SET _cnt = 1;
        WHILE _cnt <= cnt DO
                INSERT
                INTO    filler
                SELECT  _cnt;
                SET _cnt = _cnt + 1;
        END WHILE;
END
$$

DELIMITER ;

START TRANSACTION;
CALL prc_filler(1000000);
COMMIT;

INSERT
INTO    t_inner
SELECT  id,
        NULLIF(CEILING(RAND(20100527) * 100000), 100000),
        RPAD('', 200, '*')
FROM    filler;

INSERT
INTO    t_outer
SELECT  id,
        NULLIF(CEILING(RAND(20100527 << 1) * 100000), 100000),
        RPAD('', 200, '*')
FROM    filler;

There are two identical MyISAM tables. Each of the tables contains 1,000,000 random values from 1 to 99,999 and also some NULL values. There is an index on value in both tables.

Now, let’s check the queries.

NOT EXISTS

SELECT  SUM(LENGTH(stuffing)), COUNT(*)
FROM    t_outer o
WHERE   NOT EXISTS
        (
        SELECT  NULL
        FROM    t_inner i
        WHERE   i.val = o.val
        )

Field or reference '20100527_anti.o.val' of SELECT #2 was resolved in SELECT #1
select sum(length(`20100527_anti`.`o`.`stuffing`)) AS `SUM(LENGTH(stuffing))`,count(0) AS `COUNT(*)` from `20100527_anti`.`t_outer` `o` where (not(exists(select NULL from `20100527_anti`.`t_inner` `i` where (`20100527_anti`.`i`.`val` = `20100527_anti`.`o`.`val`))))

The query completes in 9.9 seconds. As we can see, it is optimized to use the index on t_inner.val and return on the first match.

LEFT JOIN / IS NULL

SELECT  SUM(LENGTH(o.stuffing)), COUNT(*)
FROM    t_outer o
LEFT JOIN
        t_inner i
ON      i.val = o.val
WHERE   i.id IS NULL

select sum(length(`20100527_anti`.`o`.`stuffing`)) AS `SUM(LENGTH(o.stuffing))`,count(0) AS `COUNT(*)` from `20100527_anti`.`t_outer` `o` left join `20100527_anti`.`t_inner` `i` on((`20100527_anti`.`i`.`val` = `20100527_anti`.`o`.`val`)) where isnull(`20100527_anti`.`i`.`id`)

The query semantics are the same as those of NOT EXISTS, and we even see the Not exists optimization in the plan, however this query performs much more poorly than NOT EXISTS: 13 seconds. Why?

MySQL documentation on EXPLAIN states that Not exists is used to optimize the queries similar to the one we have just run: LEFT JOIN with IS NULL predicate applied to a non-nullable column.

MySQL is aware that such a predicate can only be satisfied by a record resulting from a JOIN miss (i. e. when no matching record was found in the rightmost table) and stops reading records after first index hit.

However, this optimization is implemented in a way that is far from being perfect. Despite the fact that no actual value of id can be returned by such a query, the engine still looks up id in the table (since it’s not a part of the index). We can see it in the plan: unlike NOT EXISTS query, there is no Using index for t_inner. This means that a table lookup is performed.

Even we replace id with val in the query, it still performs poorly:

SELECT  SUM(LENGTH(o.stuffing)), COUNT(*)
FROM    t_outer o
LEFT JOIN
        t_inner i
ON      i.val = o.val
WHERE   i.val IS NULL

select sum(length(`20100527_anti`.`o`.`stuffing`)) AS `SUM(LENGTH(o.stuffing))`,count(0) AS `COUNT(*)` from `20100527_anti`.`t_outer` `o` left join `20100527_anti`.`t_inner` `i` on((`20100527_anti`.`i`.`val` = `20100527_anti`.`o`.`val`)) where isnull(`20100527_anti`.`i`.`val`)

This time, no table lookups are made but there is no Not exists optimization either.

Despite the fact that the join condition eliminates possibility of an actual NULL being returned by the query and any val IS NULL reaching the WHERE clause is a result of a join miss, MySQL still examines all records in t_inner, not stopping after the first hit.

This had been submitted as a bug.

Now, what about NOT IN?

NOT IN

Unlike the previous two queries that only differ in implementation, not in semantics, NOT IN, being applied as is, would yield the different results.

NOT EXISTS and IS NULL are two-state predicates, they can only return TRUE or FALSE. NOT IN is a three-state predicate: it can return TRUE, FALSE or NULL.

NULL value is returned in two cases:

• When t_outer.value being tested is NULL
• When at least one of t_inner.value is NULL

This means that having but a single NULL in t_inner would prevent the query from returning anything.

Naive approach

Let’s see what happens if we just substitute NOT IN instead of NOT EXISTS:

SELECT  SUM(LENGTH(stuffing)), COUNT(*)
FROM    t_outer o
WHERE   val NOT IN
        (
        SELECT  val
        FROM    t_inner i
        )

select sum(length(`20100527_anti`.`o`.`stuffing`)) AS `SUM(LENGTH(stuffing))`,count(0) AS `COUNT(*)` from `20100527_anti`.`t_outer` `o` where (not(<in_optimizer>(`20100527_anti`.`o`.`val`,<exists>(<index_lookup>(<cache>(`20100527_anti`.`o`.`val`) in t_inner on ix_inner_val checking NULL having trigcond(<is_not_null_test>(`20100527_anti`.`i`.`val`)))))))

Since there are NULLs in t_inner, no record in t_outer can satisfy the predicate.

MySQL does not optimize this very well. It takes but a single index scan to find out if there are NULL values in t_inner and return if they are, but for some reason MySQL still applies the condition to each record in t_outer.

Naive approach, improved

With a little help from our side, this can be improved:

SELECT  SUM(LENGTH(stuffing)), COUNT(*)
FROM    t_outer o
WHERE   NOT EXISTS
        (
        SELECT  NULL
        FROM    t_inner i
        WHERE   val IS NULL
        )
        AND val NOT IN
        (
        SELECT  val
        FROM    t_inner i
        )

select sum(length(`20100527_anti`.`o`.`stuffing`)) AS `SUM(LENGTH(stuffing))`,count(0) AS `COUNT(*)` from `20100527_anti`.`t_outer` `o` where 0

We added an explicit check for NULL values. Since it’s not correlated, MySQL could instantly prove it false, cache it and avoid the table scan at all.

Ignoring right side NULLs

Now, let’s make a NOT IN query that does not take the NULL values in t_inner into account:

SELECT  SUM(LENGTH(stuffing)), COUNT(*)
FROM    t_outer o
WHERE   val NOT IN
        (
        SELECT  val
        FROM    t_inner i
        WHERE   val IS NOT NULL
        )

select sum(length(`20100527_anti`.`o`.`stuffing`)) AS `SUM(LENGTH(stuffing))`,count(0) AS `COUNT(*)` from `20100527_anti`.`t_outer` `o` where (not(<in_optimizer>(`20100527_anti`.`o`.`val`,<exists>(<index_lookup>(<cache>(`20100527_anti`.`o`.`val`) in t_inner on ix_inner_val checking NULL where (`20100527_anti`.`i`.`val` is not null) having trigcond(<is_not_null_test>(`20100527_anti`.`i`.`val`)))))))

This time, the query returns records, but not as many as the previous queries did.

We made an additional check for NULL in t_inner but not in t_outer. There are some records in t_outer that have a NULL in val. Both IN and NOT IN would evaluate to NULL and WHERE would filter them out.

We see another glitch in MySQL optimizer here: a Full scan on NULL key applied. Since NOT IN should always return TRUE when the subquery returns no records (even if the value checked is a NULL), on correlated queries a fullscan should be applied to check for the records and find out whether to return NULL or FALSE. However, in this case the IN subquery is not correlated, so the check could only be performed once and cached, like with the LEFT JOIN.

In our case the overhead would be negligible, since the subquery would return on the first match, but it could matter if we had more NULL values in t_outer.

Now, what if we want NULL records on t_outer to be returned as well? We just need to add an additional check for NULLs.

Ignoring all NULLs

SELECT  SUM(LENGTH(stuffing)), COUNT(*)
FROM    t_outer o
WHERE   val IS NULL
        OR val NOT IN
        (
        SELECT  val
        FROM    t_inner i
        WHERE   val IS NOT NULL
        )

select sum(length(`20100527_anti`.`o`.`stuffing`)) AS `SUM(LENGTH(stuffing))`,count(0) AS `COUNT(*)` from `20100527_anti`.`t_outer` `o` where (isnull(`20100527_anti`.`o`.`val`) or (not(<in_optimizer>(`20100527_anti`.`o`.`val`,<exists>(<index_lookup>(<cache>(`20100527_anti`.`o`.`val`) in t_inner on ix_inner_val checking NULL where (`20100527_anti`.`i`.`val` is not null) having trigcond(<is_not_null_test>(`20100527_anti`.`i`.`val`))))))))

Here, the query returns the same results as NOT EXISTS.

Full scan on NULL key is still present in the plan but will never actually be executed because it will be short circuited by the previous IS NULL check.

Summary

As was shown in the earlier article, LEFT JOIN / IS NULL and NOT IN are best used to implement an anti-join in MySQL if the columns on both sides are not nullable.

The situation is different when the columns are nullable:

• NOT EXISTS performs in most straightforward way: just checks equality and returns TRUE or FALSE on the first hit / miss.
• LEFT JOIN / IS NULL either makes an additional table lookup or does not return on the first match and performs more poorly in both cases.
• NOT IN, having different semantics, requires additional checks for NULL values. These checks should be coded into the query

With nullable columns, NOT EXISTS and NOT IN (with additional checks for NULLS) are the most efficient methods to implement an anti-join in MySQL.

LEFT JOIN / IS NULL performs poorly.

SELECT  *
FROM    orders
WHERE   quantity <= 4
        AND urgency <= 4

The problem is indexing strategy, of course. Which columns should we index?

If we index quantity, the optimizer will be able to use the index to filter on it. However, filtering on urgency will require scanning all records with quantity < 4 and applying the urgency filter to each record found.

Same with urgency. We can use range access on urgency using an index, but this will require filtering on quantity.

Why, create a composite index!, some will say.

Unfortunately, that won’t help much.

A composite B-Tree index maintains what is called a lexicographical order of the records. This means that an index on (quantity, urgency) will sort on quantity, and only if the quantities are equal, it will take the urgency into account.

The picture below shows how would the records be ordered in such an index:

As we can see, with a single index range scan (i. e. just following the arrows) we cannot select only the records within the dashed rectangle. There is no single index range that could be used to filter on both columns.

Even if we changed the field order in the index, it would just change the direction of the arrows connecting the records:

, and still no single range that contains only the records we need.

Can we improve it?

If we take at closer look at the table contents we will see that despite the fact that there is no single range that contains the records we need (and only them), there are four ranges that do filter our records.

If we rewrote the query condition like that:

SELECT  *
FROM    orders
WHERE   quantity IN (0, 1, 2, 3, 4)
        AND urgency <= 4

with the first index, or like that:

SELECT  *
FROM    orders
WHERE   quantity <= 4
        AND urgency IN (0, 1, 2, 3, 4)

with the second index, then any decent SQL engine would build a decent and efficient plan (Index range in MySQL, INLIST ITERATOR in Oracle etc.)

The problem is that rewriting this query still requires an SQL developer. But this could be done automatically by the optimizer. There are several methods to do that.

Smallest superset

A range condition defines a set of the values that could possibly satisfy the condition (i. e. belong to the range). The number of the values that satisfy this condition is called set cardinality. The word number is quoted here because it can be infinite, and not all infinities are created equal: some are more infinite than the others!

However, if we take the column definition into account, we can see that some ranges define finite (and quite constrained) sets of possible values. For instance, a condition like quantity < 4 on a column that is defined as UNSIGNED INT can possibly be satisfied by five values: 0, 1, 2, 3 and 4.

This set is the smallest superset of all sets of values that satisfy the range condition.

Loose index scan

Even with the conditions that theoretically define the infinite sets of values that could satisfy them, practically there is always a finite number of values in the table that do satisfy them (the table itself contains the finite number of records, to begin with).

And most engines keep track of that number in their statistics tables: this is what is called field cardinality, a measure of field uniqueness.

If the range cardinality is expected to be low (either from the set of values that can possibly belong to the range, or from the actual number of distinct values that do belong to the range, according to statistics), it would be a wise idea to rewrite the range condition as an IN condition containing all possible values that can belong or do belong to the range.

This will replace a single less than or greater then with a small number of equals to. And an equals to gives the optimizer much more space to, um, optimize. It can use a HASH JOIN, split the index into a number of continuous ranges or do some other interesting things that can only be done with an equijoin.

With an UNSIGNED INTEGER column, it is easy to generate a set of values that could satisfy the range. But what if we know the range cardinality to be low from the statistics, not from the column datatype?

In this case, we could build the set of possible values using what MySQL calls a loose index scan.

Basically, it takes the first record from the index and then recursively searches the next lowest record whose key value is greater than the previous one, using the index seek (as opposed to index scan). This means instead of mere scanning the index and applying the condition to each field, the engine would walk up and down the B-Tree to locate the first record the the greater key. It is much more efficient when the number of distinct keys is small. And in fact, MySQL does use this method for queries that involve SELECT DISTINCT on an indexed field.

MIN / MAX

This method would be useful for the open ranges (like > or <, as opposed to BETWEEN).

By default, the INT column means SIGNED INT. For a condition like quantity <= 4 this would mean all integers from -2,147,483,647 to 4 which is way too many.

In real tables, the quantities would be something greater than 0. But not all developers bother to pick a right datatype or add CHECK constraints for their columns (and some databases like PostgreSQL lack the unsigned datatypes anyway).

To work around this, we could find the minimum existing value in the index using a single index seek. It would serve as a lower bound for the range. Since in real table that would most probably be something like 0 or 1, that would make the range much more constrained.

All these three methods could be used at the same time. Since the methods require nothing but just a single lookup of the statistics table and a single index seek, the most efficient method to return the subset of values that could satisfy the range could be chosen at runtime.

Implementation

Now, let’s make a sample table in PostgreSQL and see how could we benefit from replacing the low-cardinality ranges with the lists of values:

Table creation details

CREATE TABLE t_composite (
        id INT NOT NULL,
        uint1 INT NOT NULL,
        uint2 INT NOT NULL,
        real1 DOUBLE PRECISION NOT NULL,
        real2 DOUBLE PRECISION NOT NULL,
        stuffing VARCHAR(200) NOT NULL
        );

SELECT SETSEED(0.20100518);

INSERT
INTO    t_composite
SELECT  n,
        CEILING(RANDOM() * 40),
        CEILING(RANDOM() * 400000),
        CEILING(RANDOM() * 40) * 0.01,
        CEILING(RANDOM() * 400000) * 0.01,
        RPAD('', 200, '*')
FROM    generate_series(1, 16000000) n;

ALTER TABLE t_composite ADD CONSTRAINT pk_composite_id PRIMARY KEY (id);

CREATE INDEX ix_composite_uint ON t_composite (uint1, uint2);

CREATE INDEX ix_composite_real ON t_composite (real1, real2);

This table contains 16,000,000 records with two integer fields and two double precision fields.

There are composite indexes on the pairs of fields. This indexes are intentionally created with the least selective column leading to demonstrate the benefits of range transformation.

First, let’s run a query similar to the original one:

SELECT  SUM(LENGTH(stuffing))
FROM    t_composite
WHERE   1 = 1
        AND uint1 <= 20
        AND uint2 <= 20

Aggregate  (cost=171331.57..171331.58 rows=1 width=204)
  ->  Index Scan using ix_composite_uint on t_composite  (cost=0.00..171329.57 rows=796 width=204)
        Index Cond: ((uint1 <= 20) AND (uint2 <= 20))

The plan says that the index condition involves both fields. However, only the first field is used in the B-Tree search: the second is just being filtered on, though no actual table access is performed yet on this step.

The index keys are not too long, however, there are several millions of them that need to be scanned. That’s why the query takes more than 5 seconds to complete.

The I/O statistics show the following:

SELECT  pg_stat_reset();

SELECT  SUM(LENGTH(stuffing))
FROM    t_composite
WHERE   1 = 1
        AND uint1 <= 20
        AND uint2 <= 20;

SELECT  pg_stat_get_blocks_fetched('ix_composite_uint'::regclass);

The query required more than twenty thousands of index blocks to be read and examined

Smallest superset

Now, let’s try to substitute the hard-coded list of possible values instead of the range condition:

SELECT  SUM(LENGTH(stuffing))
FROM    t_composite
WHERE   1 = 1
        AND uint1 IN
        (
        SELECT  generate_series(0, 20)
        )
        AND uint2 <= 20

Aggregate  (cost=167.47..167.49 rows=1 width=204)
  ->  Nested Loop  (cost=0.02..167.37 rows=40 width=204)
        ->  HashAggregate  (cost=0.02..0.03 rows=1 width=4)
              ->  Result  (cost=0.00..0.01 rows=1 width=0)
        ->  Index Scan using ix_composite_uint on t_composite  (cost=0.00..166.84 rows=40 width=208)
              Index Cond: ((t_composite.uint1 = (generate_series(0, 20))) AND (t_composite.uint2 <= 20))

Instead of a giant singe range, there are 21 short ranges examined in a nested loop. This is instant (6 ms).

Let’s look into the I/O statistics again:

SELECT  pg_stat_reset();

SELECT  SUM(LENGTH(stuffing))
FROM    t_composite
WHERE   1 = 1
        AND uint1 IN
        (
        SELECT  generate_series(0, 20)
        )
        AND uint2 <= 20;

SELECT  pg_stat_get_blocks_fetched('ix_composite_uint'::regclass);

Now, only 64 blocks need to be read.

MIN / MAX

We took 0 as the initial value, but since theoretically there can be negative numbers in the columns, this assumption is not safe.

We need to get the least value from the table instead of assuming it:

SELECT  SUM(LENGTH(stuffing))
FROM    t_composite
WHERE   1 = 1
        AND uint1 IN
        (
        SELECT  generate_series(
                (
                SELECT  MIN(uint1)
                FROM    t_composite
                ), 20)
        )
        AND uint2 <= 20

Aggregate  (cost=171.40..171.41 rows=1 width=204)
  ->  Nested Loop  (cost=3.95..171.29 rows=40 width=204)
        ->  HashAggregate  (cost=3.95..3.96 rows=1 width=4)
              ->  Result  (cost=3.92..3.93 rows=1 width=0)
                    InitPlan 2 (returns $1)
                      ->  Result  (cost=3.91..3.92 rows=1 width=0)
                            InitPlan 1 (returns $0)
                              ->  Limit  (cost=0.00..3.91 rows=1 width=4)
                                    ->  Index Scan using ix_composite_uint on t_composite  (cost=0.00..62571556.68 rows=15999664 width=4)
                                          Filter: (uint1 IS NOT NULL)
        ->  Index Scan using ix_composite_uint on t_composite  (cost=0.00..166.84 rows=40 width=208)
              Index Cond: (("20100519_cardinality".t_composite.uint1 = (generate_series($1, 20))) AND ("20100519_cardinality".t_composite.uint2 <= 20))

This is instant again.

Now, what about statistics?

SELECT  pg_stat_reset();

SELECT  SUM(LENGTH(stuffing))
FROM    t_composite
WHERE   1 = 1
        AND uint1 IN
        (
        SELECT  generate_series(
                (
                SELECT  MIN(uint1)
                FROM    t_composite
                ), 20)
        )
        AND uint2 <= 20;

SELECT  pg_stat_get_blocks_fetched('ix_composite_uint'::regclass);

The query is only 3 block reads heavier, but this time it is guaranteed to be correct.

Loose index scan

The real* columns hold the double precision data.

First, let’s run the original query:

SELECT  SUM(LENGTH(stuffing))
FROM    t_composite
WHERE   1 = 1
        AND real1 <= 0.2
        AND real2 <= 0.2

Aggregate  (cost=206210.82..206210.84 rows=1 width=204)
  ->  Index Scan using ix_composite_real on t_composite  (cost=0.00..206208.84 rows=793 width=204)
        Index Cond: ((real1 <= 0.2::double precision) AND (real2 <= 0.2::double precision))

Again, there are way too many block reads:

SELECT  pg_stat_reset();

SELECT  SUM(LENGTH(stuffing))
FROM    t_composite
WHERE   1 = 1
        AND real1 <= 0.2
        AND real2 <= 0.2;

SELECT  pg_stat_get_blocks_fetched('ix_composite_real'::regclass);

, and the query takes more than 6 seconds.

For a condition like real1 <= 0.2, the smallest superset of all possible values (that is all possible double-precision values between 0 and 0.2) would be too large (though still finite of course) to be generated and joined. That’s why we need to use server-collected statistics to decide whether a loose index scan would be efficient to get the list of all distinct values of real1 in the table:

SELECT  n_distinct, most_common_vals, histogram_bounds
FROM    pg_stats
WHERE   schemaname = '20100519_cardinality'
        AND tablename = 't_composite'
        AND attname IN ('real1', 'real2')

From the server statistics we see that there are only 40 distinct values of real1. Hence, a loose index scan as such would be efficient.

Let’s look into the stats on real2. We see that there are 362181 distinct values in the table, and the range in question (<= 0.2) corresponds to the first entry in the histogram (BETWEEN 0.07 AND 44.3). Since the histogram splits the values into 100 percentiles, this means that there are about 3622 values from 0.07 to 44.3, and ((0.2 - 0.07) / (44.3 - 0.07) * 362181 / 100) ≈ 11 distinct values inside our range, and about 11 * 16000000 /362181 ≈ 486 records with these values.

Taking into account that the original query would need to scan 8,000,000 records, the loose index scan seems to be a good idea.

Unfortunately, PostgreSQL does not support it directly, but with minimal effort it can be emulated:

WITH    RECURSIVE q (real1) AS
        (
        SELECT  MIN(real1)
        FROM    t_composite
        UNION ALL
        SELECT  (
                SELECT  c.real1
                FROM    t_composite c
                WHERE   c.real1 > q.real1
                        AND c.real1 <= 0.2
                ORDER BY
                        c.real1
                LIMIT 1
                )
        FROM    q
        WHERE   q.real1 IS NOT NULL
        )
SELECT  SUM(LENGTH(stuffing))
FROM    q
JOIN    t_composite c
ON      c.real1 = q.real1
        AND c.real2 <= 0.2

Aggregate  (cost=17319.78..17319.80 rows=1 width=204)
  CTE q
    ->  Recursive Union  (cost=3.92..401.33 rows=101 width=8)
          ->  Result  (cost=3.92..3.93 rows=1 width=0)
                InitPlan 1 (returns $1)
                  ->  Limit  (cost=0.00..3.92 rows=1 width=8)
                        ->  Index Scan using ix_composite_real on t_composite  (cost=0.00..62694798.76 rows=15999664 width=8)
                              Filter: (real1 IS NOT NULL)
          ->  WorkTable Scan on q  (cost=0.00..39.54 rows=10 width=8)
                Filter: (q.real1 IS NOT NULL)
                SubPlan 2
                  ->  Limit  (cost=0.00..3.93 rows=1 width=8)
                        ->  Index Scan using ix_composite_real on t_composite c  (cost=0.00..314695.34 rows=79998 width=8)
                              Index Cond: ((real1 > $2) AND (real1 <= 0.2::double precision))
  ->  Nested Loop  (cost=0.00..16914.48 rows=1588 width=204)
        ->  CTE Scan on q  (cost=0.00..2.02 rows=101 width=8)
        ->  Index Scan using ix_composite_real on t_composite c  (cost=0.00..166.95 rows=40 width=212)
              Index Cond: ((c.real1 = q.real1) AND (c.real2 <= 0.2::double precision))

This is almost instant again. Here are the I/O statistics:

SELECT  pg_stat_reset();

WITH    RECURSIVE q (real1) AS
        (
        SELECT  MIN(real1)
        FROM    t_composite
        UNION ALL
        SELECT  (
                SELECT  c.real1
                FROM    t_composite c
                WHERE   c.real1 > q.real1
                        AND c.real1 <= 0.2
                ORDER BY
                        c.real1
                LIMIT 1
                )
        FROM    q
        WHERE   q.real1 IS NOT NULL
        )
SELECT  SUM(LENGTH(stuffing))
FROM    q
JOIN    t_composite c
ON      c.real1 = q.real1
        AND c.real2 <= 0.2;

SELECT  pg_stat_get_blocks_fetched('ix_composite_real'::regclass);

The number of index reads is reduced greatly again.

Summary

In some cases, a range predicate (like less than, greater than or between) can be rewritten as an IN predicate against the list of values that could satisfy the range condition.

Depending on the column datatype, check constraints and statistics, that list could be comprised of all possible values defined by the column’s domain; all possible values defined by column’s minimal and maximal value, or all actual distinct values contained in the table. In the latter case, a loose index scan could be used to retrieve the list of such values.

Since an equality condition is applied to each value in the list, more access and join methods could be used to build the query plain, including range conditions on secondary index columns, hash lookups etc.

Whenever the optimizer builds a plan for a query that contains a range predicate, it should consider rewriting the range condition as an IN predicate and use the latter method if it proves more efficient.

Ivo Radev asks:

I am trying to make a very simple query.

We have a log table which different machines write to. Given the machine list, I need to find the latest log timestamp.

Currently, the query looks like this:

SELECT  MAX(log_time)
FROM    log_table
WHERE   log_machine IN ($machines)

, and I pass the comma-separated list of $machines from PHP.

The weird thing is that the query is literally instant when there is only one machine (any) in the list but slow when there are multiple machines.

I’m considering doing it in separate queries and then process the results in PHP. However I’d like to know if there is a fast solution in MySQL.

Most probably, there is a composite index on (log_machine, log_time) which is being used for the query.

Usually, a query like this:

SELECT  MAX(log_time)
FROM    log_table

on the indexed field log_time can be served with a single index seek on the index.

Indeed, the MAX(log_time), by definition, is the latest entry in the index order, and can be fetched merely by finding the trailing index entry. It’s a matter of several page reads in the B-Tree, each one following the rightmost link to the lower-level page.

Similarly, this query:

SELECT  MAX(log_time)
FROM    log_table
WHERE   log_machine = $my_machine

can be served with a single index seek too. However, the index should include log_machine as a leading column.

In this case, a set of records satisfying the WHERE clause of the query is represented by a single logically continuous block of records in the index, each one sharing the same value of log_machine. MAX(log_time) will of course be held by the last record in this block. MySQL just finds that last record and takes the log_time out of it.

Now, what if we have a multiple condition on log_machine?

The index remains the same, but the record holding MAX(log_time) is not the last record in a single continuous block anymore. Instead, there are multiple blocks each having its own MAX(log_date). log_time cannot be found merely by taking the last record from the index block: it is not known which one is the correct one.

On composite indexes, however, MySQL offers loose index scan. This means that it jumps over the distinct values of the leading column, doing an index seek (instead of index scan) to retrieve each next value.

As stated in the documentation, this method is ideal to doing the queries like that:

SELECT  log_machine, MAX(log_time)
FROM    log_table
WHERE   log_machine IN ($my_machine_list)

As we said earlier, for each log_machine, its MAX(log_time) can be returned very fast, and the list of the log_machines could be obtained with a loose index scan, by seeking the keys in the index.

This query, however, will not produce a single MAX(log_time): instead, it will return as many maximums as there are values in the list (which are found in the table, of course).

But this can be easily worked around: we just select the greatest one of these records. Since the subquery will only return several records, the greatest one if them can be found almost instantly.

Let’s create a sample table:

Table creation details

CREATE TABLE filler (
        id INT NOT NULL PRIMARY KEY AUTO_INCREMENT
) ENGINE=Memory;

CREATE TABLE log_table (
        id INT NOT NULL PRIMARY KEY,
        log_machine VARCHAR(20) NOT NULL,
        log_time DATETIME NOT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

DELIMITER $$

CREATE PROCEDURE prc_filler(cnt INT)
BEGIN
        DECLARE _cnt INT;
        SET _cnt = 1;
        WHILE _cnt <= cnt DO
                INSERT
                INTO    filler
                SELECT  _cnt;
                SET _cnt = _cnt + 1;
        END WHILE;
END
$$

DELIMITER ;

START TRANSACTION;
CALL prc_filler(1000000);
COMMIT;

INSERT
INTO    log_table
SELECT  id,
        CONCAT('Machine ' , CEILING(RAND(20100508) * 10)),
        '2010-05-08' - INTERVAL CEILING(RAND(20100508 << 1) * 10000000) SECOND
FROM    filler f1;

CREATE INDEX ix_log_machine_time ON log_table (log_machine, log_time);

The table has 1,000,000 records

Now, let’s see how the original query performs:

SELECT  MAX(log_time) AS maxtime
FROM    log_table
WHERE   log_machine IN ('Machine 3', 'Machine 5', 'Machine 7', 'Machine 9')

select max(`20100508_max`.`log_table`.`log_time`) AS `maxtime` from `20100508_max`.`log_table` where (`20100508_max`.`log_table`.`log_machine` in ('Machine 3','Machine 5','Machine 7','Machine 9'))

The query uses range access to retrieve the records and browses all records to find the maximum. It takes 640 ms on a table 1,000,000 log records (which is about a day’s output of a single web server under a load decent but not super hard).

Now, let’s try to select the greatest of the group-wise maximums:

SELECT  MAX(log_time) AS maxtime
FROM    log_table
WHERE   log_machine IN ('Machine 3', 'Machine 5', 'Machine 7', 'Machine 9')
GROUP BY
        log_machine
ORDER BY
        1 DESC
LIMIT 1

select max(`20100508_max`.`log_table`.`log_time`) AS `maxtime` from `20100508_max`.`log_table` where (`20100508_max`.`log_table`.`log_machine` in ('Machine 3','Machine 5','Machine 7','Machine 9')) group by `20100508_max`.`log_table`.`log_machine` order by 1 desc limit 1

Now, it’s instant, as it should be.

As it often happens, by appending three seemingly redundant clauses to a query we made MySQL to choose a more efficient plan and the query is now instant even with multiple machines in the list.

Hope that helps.

I’m always glad to answer the questions regarding database queries.

Ask me a question

This algorithm operates on two sorted recordsets, keeping two pointers that chase each other.

The Wikipedia entry above describes it quite well in terms of algorithms. I’ll just make an animated GIF to make it more clear:

This is quite a nice and elegant algorithm, which, unfortunately, has two major drawbacks:

1. It needs the recordsets to be sorted
2. Even with the recordsets sorted, it is no better than a HASH JOIN

The sorting part is essential for this algorithm and there is nothing that can be done with it: the recordsets should be sorted, period. Databases, however, often provide the records in the sorted order: from clustered tables, indexes, previously sorted and ordered subqueries, spool tables etc.

But even when the recordsets are already sorted, on equijoins the MERGE JOIN is hardly faster than a HASH JOIN.

Why?

MERGE JOIN vs. HASH JOIN

Let’s remember how the HASH JOIN works:

• It takes the smaller table and builds a hash table out of it, with the join key as the hash key.
• Then it takes each record from the larger table and looks it up in the hash table. If found, the records are returned.

We see that there are four major steps involved:

1. Scan the smaller table
2. Build a hash table (i. e. copy each record from the smaller table into the hash slot)
3. Scan the larger table
4. Look up the larger table

Since building and looking up the hash table are performed in memory (or, depending on the SQL engine implementation, in memory-mapped temporary database, which is almost the same), these steps take negligible time compared to the time required to scan the table.

But we see that MERGE JOIN, as it is implemented now, also requires scanning both recordsets. Each record should be evaluated by the pointer to figure out if its join key is more, less or equal to that of the the other pointer.

This means that both MERGE JOIN and HASH JOIN require scanning both recordsets. However, HASH JOIN does not require any special order, which means it can use a table scan, index fast full scan and any other methods to get the records all at once, while MERGE JOIN need either to sort the records (which is obviously slow) or to traverse the index with the subsequent key lookups (which is not fast too).

In some terminal cases MERGE JOIN can be more efficient indeed: say, when the hash table does not fit completely into memory and would require either extensive disk writes or several scans over the source tables, while a MERGE JOIN could be performed on a pair of indexes.

It is also efficient for FULL OUTER JOIN: each record is evaluated, returned and forgotten only once, while HASH JOIN would require a second pass over the records that had not been ever matched.

Seeks instead of scans

But does the MERGE JOIN really always need to traverse all records?

Let’s see some more pictures:

Here, the right recordset is 100,000 records ahead of the left recordset. With MERGE JOIN, 100,000 records should be scanned from the left recordset and 100,000 comparisons made.

This is unavoidable if the recordset is a result of a sort operation.

However, MERGE JOIN is usually chosen when there is a more efficient sorted row source available: an index or a spool table (temporary index built in runtime). And both these sources allow efficient random seeks.

If an index served as the left recordset, we could see that right pointer is too far ahead, and just seek for its value in the left recordset instead of scanning 100,000 records:

Here, we can see that 100,000 is too far away and could advance the left pointer to the position of the right pointer in only several reads, traversing the B-Tree.

Since the indexes usually collect statistics, all we would need to do to decide whether we need to seek or scan was to check the histograms to estimate how may records are there between the current and the opposite pointers. If there are too many, the seek cost would overweight the scan cost and a seek should be performed. The statistics table itself would not need to be queried too often: since the records are always selected in order, the statistics table could be also read sequentially.

Emulation

Let’s create a couple of PostgreSQL tables and see the performance benefit:

Table creation details

CREATE TABLE t_left (
        id INT NOT NULL PRIMARY KEY,
        good INT NOT NULL,
        bad INT NOT NULL,
        stuffing VARCHAR(200) NOT NULL
);

INSERT
INTO    t_left
SELECT  s, s, s, RPAD('', 200, '*')
FROM    generate_series(1, 1000000) s;

CREATE UNIQUE INDEX ix_left_good ON t_left (good);

CREATE UNIQUE INDEX ix_left_bad ON t_left (bad);

CREATE TABLE t_right (
        id INT NOT NULL PRIMARY KEY,
        good INT NOT NULL,
        bad INT NOT NULL,
        stuffing VARCHAR(200) NOT NULL
);

INSERT
INTO    t_right
SELECT  s, s, s + 999000, RPAD('', 200, '*')
FROM    generate_series(1, 1000000) s;

CREATE UNIQUE INDEX ix_right_good ON t_right (good);

CREATE UNIQUE INDEX ix_right_bad ON t_right (bad);

These two tables have 1,000,000 records each, and a common column that would return only 1,000 records in a join.

Here’s the plain query runs against these tables:

SELECT  SUM(LENGTH(l.stuffing) + LENGTH(r.stuffing))
FROM    t_left l
JOIN    t_right r
ON      r.bad = l.bad

Aggregate  (cost=71338.33..71338.35 rows=1 width=408)
  ->  Merge Join  (cost=58737.16..68838.33 rows=1000000 width=408)
        Merge Cond: (l.bad = r.bad)
        ->  Index Scan using ix_left_bad on t_left l  (cost=0.00..56287.36 rows=1000000 width=208)
        ->  Index Scan using ix_right_bad on t_right r  (cost=0.00..56287.36 rows=1000000 width=208)

Note that PostgreSQL used a MERGE JOIN without any tricks from our side. This is because the table records are too large and could not fit into a hash table all at once.

Of course, PostgreSQL could only store the record pointers in the hash table and do the record lookups after the join, however, for some reason it would not select this plan.

MERGE JOIN, in our case, is quite efficient, since the indexes are read first and the actual records are only looked up for the matched records (which are not too numerous). However, it still requires traversing 2,000,000 records which takes more than a second.

Now, let’s emulate the MERGE JOIN doing the seeks instead of scans. To do that, we will write a recursive query:

WITH    RECURSIVE q (l, r) AS
        (
        SELECT  (
                SELECT  l
                FROM    t_left l
                ORDER BY
                        bad
                LIMIT 1
                ),
                (
                SELECT  r
                FROM    t_right r
                ORDER BY
                        bad
                LIMIT 1
                )
        UNION ALL
        SELECT  CASE
                WHEN (q.l).bad < (q.r).bad THEN
                        (
                        SELECT  li
                        FROM    t_left li
                        WHERE   li.bad >= (q.r).bad
                        ORDER BY
                                bad
                        LIMIT 1
                        )
                WHEN (q.l).bad = (q.r).bad THEN
                        (
                        SELECT  li
                        FROM    t_left li
                        WHERE   li.bad > (q.r).bad
                        ORDER BY
                                bad
                        LIMIT 1
                        )
                ELSE
                        l
                END,
                CASE
                WHEN (q.r).bad < (q.l).bad THEN
                        (
                        SELECT  ri
                        FROM    t_right ri
                        WHERE   ri.bad >= (q.l).bad
                        ORDER BY
                                bad
                        LIMIT 1
                        )
                WHEN (q.r).bad = (q.l).bad THEN
                        (
                        SELECT  ri
                        FROM    t_right ri
                        WHERE   ri.bad > (q.l).bad
                        ORDER BY
                                bad
                        LIMIT 1
                        )
                ELSE
                        r
                END
        FROM    q
        WHERE   l IS NOT NULL
                AND r IS NOT NULL
        )
SELECT  SUM(LENGTH((q.l).stuffing) + LENGTH((q.r).stuffing))
FROM    q
WHERE   (q.l).bad = (q.r).bad

Aggregate  (cost=30.94..30.96 rows=1 width=64)
  CTE q
    ->  Recursive Union  (cost=0.11..28.66 rows=101 width=64)
          ->  Result  (cost=0.11..0.12 rows=1 width=0)
                InitPlan 1 (returns $1)
                  ->  Limit  (cost=0.00..0.06 rows=1 width=36)
                        ->  Index Scan using ix_left_bad on t_left l  (cost=0.00..56287.36 rows=1000000 width=36)
                InitPlan 2 (returns $2)
                  ->  Limit  (cost=0.00..0.06 rows=1 width=36)
                        ->  Index Scan using ix_right_bad on t_right r  (cost=0.00..56287.36 rows=1000000 width=36)
          ->  WorkTable Scan on q  (cost=0.00..2.65 rows=10 width=64)
                Filter: ((q.l IS NOT NULL) AND (q.r IS NOT NULL))
                SubPlan 3
                  ->  Limit  (cost=0.00..0.06 rows=1 width=36)
                        ->  Index Scan using ix_left_bad on t_left li  (cost=0.00..19598.69 rows=333333 width=36)
                              Index Cond: (bad >= ($3).bad)
                SubPlan 4
                  ->  Limit  (cost=0.00..0.06 rows=1 width=36)
                        ->  Index Scan using ix_left_bad on t_left li  (cost=0.00..19598.69 rows=333333 width=36)
                              Index Cond: (bad > ($3).bad)
                SubPlan 5
                  ->  Limit  (cost=0.00..0.06 rows=1 width=36)
                        ->  Index Scan using ix_right_bad on t_right ri  (cost=0.00..19598.69 rows=333333 width=36)
                              Index Cond: (bad >= ($4).bad)
                SubPlan 6
                  ->  Limit  (cost=0.00..0.06 rows=1 width=36)
                        ->  Index Scan using ix_right_bad on t_right ri  (cost=0.00..19598.69 rows=333333 width=36)
                              Index Cond: (bad > ($4).bad)
  ->  CTE Scan on q  (cost=0.00..2.27 rows=1 width=64)
        Filter: ((l).bad = (r).bad)

This query makes a seek each time it needs to advance a pointer. This is not the most efficient way, but despite that fact, this query completes in only 40 ms, which is 25 times as fast as the plain MERGE JOIN query.

Summary

With its current implementation, MERGE JOIN is not the most efficient algorithm, however, for several types of queries it outperforms HASH JOIN.

The main drawback of the MERGE JOIN is its inability to use seeks to advance the record pointers. Even if the opposite pointer is far away, the sequential scan is used instead of a B-Tree seek, even if the recordset is an index or a spool table.

To improve this, the accumulated index statistics should be taken into account when deciding whether to perform a seek or a sequential scan to catch up with the opposite pointer. If the statistics show a high number of the records in between, an index seek should be used instead of the index scan.

With this improvement, MERGE JOIN would perform much better, especially when joining two large indexed tables. It would require much less resources than a HASH JOIN, and, unlike NESTED LOOPS, the seeks would be performed only when really needed, thus preserving the benefits of the sequential access to the tables.

Nate asks:

I know you’ve addressed similar issues related to the greatest-per-group query but this seems to be a different take on that.

Example table:

t_group

item_id	group_id	score
100	1	2
100	2	3
200	1	1
300	1	4
300	2	2

Each item may be in multiple groups. Each instance of an item in that group is given a score (how relevant it is the the group).

So given the data above, when querying for group 1 it should return items 200 and 300 (item 100‘s highest score is for group 2, so it’s excluded).

The classical greatest-n-per-group problem requires selecting a single record from each group holding a group-wise maximum. This case is a little bit different: for a given group, we need to select all records holding an item-wise maximum.

Let’s create a sample table:

Table creation details

CREATE TABLE filler (
        id INT NOT NULL PRIMARY KEY AUTO_INCREMENT
) ENGINE=Memory;

CREATE TABLE t_groups (
        item_id INT NOT NULL,
        group_id INT NOT NULL,
        score INT NOT NULL,
        PRIMARY KEY (group_id, item_id),
        KEY ix_groups_gsi (item_id, score, group_id)
) ENGINE=InnoDB;              

DELIMITER $$

CREATE PROCEDURE prc_filler(cnt INT)
BEGIN
        DECLARE _cnt INT;
        SET _cnt = 1;
        WHILE _cnt <= cnt DO
                INSERT
                INTO    filler
                SELECT  _cnt;
                SET _cnt = _cnt + 1;
        END WHILE;
END
$$

DELIMITER ;

START TRANSACTION;
CALL prc_filler(1000000);
COMMIT;

INSERT
INTO    t_groups
SELECT  (id - 1) % 1000 + 1,
        (id - 1) div 1000 + 1,
        CEILING(RAND(20100422) * 10000)
FROM    filler;

This table contains 1,000,000 records: 1,000 items in 1,000 groups with random scores.

Let’s write a query which would return us all items whose largest score is in group 1.

To do this, we need to select all items from group 1 and check that no other group has a greater value of score for that item. The most intuitive query for this would look like that:

SELECT  *
FROM    t_groups go
WHERE   group_id = 1
        AND NOT EXISTS
        (
        SELECT  group_id
        FROM    t_groups gi
        WHERE   gi.item_id = go.item_id
                AND (gi.score, gi.group_id) > (go.score, go.group_id)
        )

Field or reference '20100422_rank.go.item_id' of SELECT #2 was resolved in SELECT #1
Field or reference '20100422_rank.go.score' of SELECT #2 was resolved in SELECT #1
Field or reference '20100422_rank.go.group_id' of SELECT #2 was resolved in SELECT #1
select `20100422_rank`.`go`.`item_id` AS `item_id`,`20100422_rank`.`go`.`group_id` AS `group_id`,`20100422_rank`.`go`.`score` AS `score` from `20100422_rank`.`t_groups` `go` where ((`20100422_rank`.`go`.`group_id` = 1) and (not(exists(select `20100422_rank`.`gi`.`group_id` AS `group_id` from `20100422_rank`.`t_groups` `gi` where ((`20100422_rank`.`gi`.`item_id` = `20100422_rank`.`go`.`item_id`) and ((`20100422_rank`.`gi`.`score`,`20100422_rank`.`gi`.`group_id`) > (`20100422_rank`.`go`.`score`,`20100422_rank`.`go`.`group_id`)))))))

This query returns us 3 items and their score. We see that these items are not ranked higher in any other group.

However, this query is quite inefficient: it takes almost half a second. This is way too long for 1K items. Let’s look how can it be optimized.

If we look into the query plan we will see that no range access path is used for the subquery, despite the condition that can easily be optimized used such an access path. Instead, MySQL uses only the ref access on item_id, combined with Using where; Using index.

What does it all mean?

We have the composite index on (item_id, score, group_id). This means that within the index, the records are ordered first on item_id then by score and then, in a case of a tie, by group_id.

For group 1, MySQL should perform 1,000 comparisons: for each item within the group, the engine should make sure that no other group has a higher score for the same item.

Ideally, MySQL should have found each given set of (item_id, score, group_id) in the index and then just make a single next key search to check if this record is last in the index within the given item_id. That would show as a range search in the query plan, since we actually are checking values between (item_id, score, group_id) and (item_id, +INF, +INF).

However, MySQL cannot do such things in a correlated subquery. Instead, it uses the ref access path: takes all records with the given item_id (i. e. between (item_id, -INF, -INF) and (item_id, +INF, +INF)) and traverses them applying the WHERE filter to each record.

For group 1, only 3 items are returned. This means that for majority of items (997 items), the whole index range for the items (1,000 records per item) had to be scanned.

And since the matching record is always the last one in the index range (because it’s the one with the greatest score), this means that even for returned items, the whole ranged had to be scanned too. The only difference is that the last record in the range satisfies the WHERE condition. No wonder it takes so long.

To speed up the query we need to trick MySQL a little.

Since it’s always the last record in the index range that satisfies the WHERE condition, why don’t we just take it and compare? If we see that it holds our group, we return TRUE, if we don’t, we return FALSE right away.

Here’s how we can do this:

SELECT  *
FROM    t_groups go
WHERE   group_id = 1
        AND group_id =
        (
        SELECT  group_id
        FROM    t_groups gi
        WHERE   gi.item_id = go.item_id
        ORDER BY
                item_id DESC, score DESC, group_id DESC
        LIMIT 1
        )

Field or reference '20100422_rank.go.item_id' of SELECT #2 was resolved in SELECT #1
select `20100422_rank`.`go`.`item_id` AS `item_id`,`20100422_rank`.`go`.`group_id` AS `group_id`,`20100422_rank`.`go`.`score` AS `score` from `20100422_rank`.`t_groups` `go` where ((`20100422_rank`.`go`.`group_id` = 1) and (1 = (select `20100422_rank`.`gi`.`group_id` AS `group_id` from `20100422_rank`.`t_groups` `gi` where (`20100422_rank`.`gi`.`item_id` = `20100422_rank`.`go`.`item_id`) order by `20100422_rank`.`gi`.`item_id` desc,`20100422_rank`.`gi`.`score` desc,`20100422_rank`.`gi`.`group_id` desc limit 1)))

Instead of checking the existence of the last row in a range, we check it’s value. This means that exactly one index record will be evaluated for each item within the group.

Note that the execution plan used by this query looks exactly the same as the first query’s one. Same ref condition on item_id, same Using where; Using index. The only difference (not shown in the EXPLAIN output) it that the index is traversed in descending order now. Instead of fetching 1,000 records from the beginning to check the existence of the last one, we just take one record from the end and check its value.

Those who read my blog already know why do we use seemingly redundant item_id DESC in the ORDER BY clause here. To make MySQL to use descending index access path instead of a filesort in an ordered query, we should list all clauses that constitute the index, even if some of them are filtered by the WHERE condition.

The second query completes in only 10 ms which is 40 times as fast as the original query.

Hope that helps.

I’m always glad to answer the questions regarding database queries.

Ask me a question

James asks:

Your series on hierarchical queries in MySQL is tremendous! I’m using it to create a series of threaded conversations.

I’m wondering if there is a way to paginate these results.

Specifically, let’s say I want to limit the conversations to return 10 root nodes (parent=0) and all of their children in a query.

I can’t just limit the final query, because that will clip off children. I’ve tried to add LIMITs to your stored functions, but I’m not getting the magic just right.

How would you go about doing this?

A quick reminder: MySQL does not support recursion (either CONNECT BY style or recursive CTE style), so using an adjacency list model is a somewhat complicated task.

However, it is still possible. The main idea is storing the recursion state in a session variable and call a user-defined function repeatedly to iterate over the tree, thus emulating recursion. The article mentioned in the question shows how to do that.

Normally, reading and assigning session variables in the same query is discouraged in MySQL, since the order of evaluation is not guaranteed. However, in the case we only use the table as a dummy recordset and no values of the records are actually used in the function, so the actual values returned by the function are completely defined by the function itself. The table is only used to ensure that the function is called enough times, and to present its results in form of a native resultset (which can be returned or joined with).

To do something with the logic of the function (like, imposing a limit on the parent nodes without doing the same on the child nodes), we, therefore, should tweak the function code, not the query that calls the functions. The only thing that matters in such a query is the number of records returned and we don’t know it in design time.

Limiting the parent nodes is quite simple: we just use another session variable to track the number of parent branches yet to be returned and stop processing as soon as the limit is hit, that is the variable becomes zero.

Let’s create a sample table and see how to do this:

Table creation details

CREATE TABLE filler (
        id INT NOT NULL PRIMARY KEY AUTO_INCREMENT
) ENGINE=Memory;

CREATE TABLE t_hierarchy (
        id int(10) unsigned NOT NULL AUTO_INCREMENT,
        parent int(10) unsigned NOT NULL,
        root INT NOT NULL,
        PRIMARY KEY (id),
        KEY ix_hierarchy_parent (parent, id),
        KEY ix_hierarchy_root (root)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

DELIMITER $$

CREATE PROCEDURE prc_filler(cnt INT)
BEGIN
        DECLARE _cnt INT;
        SET _cnt = 1;
        WHILE _cnt <= cnt DO
                INSERT
                INTO    filler
                SELECT  _cnt;
                SET _cnt = _cnt + 1;
        END WHILE;
END
$$

DELIMITER ;

START TRANSACTION;
CALL prc_filler(100000);
COMMIT;

INSERT
INTO    t_hierarchy
SELECT  id,
        CASE (id - 1) % 8
        WHEN 0 THEN 0
        ELSE FLOOR((((id - 1) % 8) + 1) / 2) + ((id - 1) div 8) * 8
        END,
        ((id - 1) div 8) * 8 + 1
FROM    filler;

There are 100,000 hierarchical records in multiple trees (8 records in each tree).

To limit the number of trees returned, we create the function similar to the one created in the earlier posts and add a little condition that would decrease the session variable, @parent_limit, each time a parent entry is returned. When this variable hits zero, it’s a signal to stop processing the records:

CREATE FUNCTION hierarchy_connect_by_parent_eq_prior_id(value INT) RETURNS INT
NOT DETERMINISTIC
READS SQL DATA
BEGIN
        DECLARE _id INT;
        DECLARE _parent INT;
        DECLARE _next INT;
        DECLARE CONTINUE HANDLER FOR NOT FOUND SET @id = NULL;

        SET _parent = @id;
        SET _id = -1;

        IF @id IS NULL THEN
                RETURN NULL;
        END IF;

        LOOP
                SELECT  MIN(id)
                INTO    @id
                FROM    t_hierarchy
                WHERE   parent = _parent
                        AND id > _id;
                IF @id IS NOT NULL OR _parent = @start_with THEN
                        SET @level = @level + 1;
                        IF _parent = @start_with AND @parent_limit > 0 THEN
                                SET @parent_limit = @parent_limit - 1;
                        END IF;
                        IF @parent_limit = 0 THEN
                                SET @id = NULL;
                        END IF;
                        RETURN @id;
                END IF;
                SET @level := @level - 1;
                SELECT  id, parent
                INTO    _id, _parent
                FROM    t_hierarchy
                WHERE   id = _parent;
        END LOOP;
END;

Let’s check it:

SELECT  CONCAT(REPEAT('    ', level - 1), CAST(hi.id AS CHAR)) AS treeitem, parent, level, lmt
FROM    (
        SELECT  hierarchy_connect_by_parent_eq_prior_id(id) AS id, @level AS level, @parent_limit as lmt
        FROM    (
                SELECT  @start_with := 0,
                        @parent_limit := 4,
                        @id := @start_with,
                        @level := 0
                ) vars, t_hierarchy
        WHERE   @id IS NOT NULL
        ) ho
JOIN    t_hierarchy hi
ON      hi.id = ho.id

select concat(repeat('    ',(`ho`.`level` - 1)),cast(`20100418_limit`.`hi`.`id` as char charset latin1)) AS `treeitem`,`20100418_limit`.`hi`.`parent` AS `parent`,`ho`.`level` AS `level`,`ho`.`lmt` AS `lmt` from (select `hierarchy_connect_by_parent_eq_prior_id`(`20100418_limit`.`t_hierarchy`.`id`) AS `id`,(@level) AS `level`,(@parent_limit) AS `lmt` from (select (@start_with:=0) AS `@start_with := 0`,(@parent_limit:=4) AS `@parent_limit := 4`,(@id:=(@start_with)) AS `@id := @start_with`,(@level:=0) AS `@level := 0`) `vars` join `20100418_limit`.`t_hierarchy` where ((@id) is not null)) `ho` join `20100418_limit`.`t_hierarchy` `hi` where (`20100418_limit`.`hi`.`id` = `ho`.`id`)

The first 3 branches in a proper hierarchy, almost instantly. Note that we need set @parent_limit to 4, i. e. one greater than the value we need.

Note that it is possible to achieve the similar behavior without using any recursive functions at all.

On most forums the filtering is performed on the topic starters, so it’s often a good idea to store the id of the topic starter along with each reply. In the sample table a did that as well.

With this, the filtering required becomes very simple:

SELECT  h.*
FROM    (
        SELECT  id
        FROM    t_hierarchy
        WHERE   parent = 0
        ORDER BY
                id
        LIMIT 3
        ) q
JOIN    t_hierarchy h
ON      h.root = q.id

select `20100418_limit`.`h`.`id` AS `id`,`20100418_limit`.`h`.`parent` AS `parent`,`20100418_limit`.`h`.`root` AS `root` from (select `20100418_limit`.`t_hierarchy`.`id` AS `id` from `20100418_limit`.`t_hierarchy` where (`20100418_limit`.`t_hierarchy`.`parent` = 0) order by `20100418_limit`.`t_hierarchy`.`id` limit 3) `q` join `20100418_limit`.`t_hierarchy` `h` where (`20100418_limit`.`h`.`root` = `q`.`id`)

This solution (that had been used on numerous forum engines for ages) is more efficient, since no function calls are involved, and more simple too.

However, it does not preserve the hierarchical order and does not allow sorting on anything but the topic starter, so if you need anything of these, the recursive function is still a way to go.

Hope that helps.

I’m always glad to answer the questions regarding database queries.

Ask me a question