I have a logfile which logs the insert/delete/updates from all kinds of tables.

I would like to get an overview of for example the last 20 people which records where updated, ordered by the last update (datetime DESC)

A common solution for such a task would be writing an aggregate query with ORDER BY and LIMIT:

SELECT  person, MAX(ts) AS last_update
FROM    logfile
GROUP BY
        person
ORDER BY
        last_update DESC
LIMIT 20

What’s bad in this solution? Performance, as usual.

Since last_update is an aggregate, it cannot be indexed. And ORDER BY on unindexed fields results in our good old friend, filesort.

Note that even in this case the indexes can be used and the full table scan can be avoided: if there is an index on (person, ts), MySQL will tend to use a loose index scan on this index, which can save this query if there are relatively few persons in the table. However, if there are many (which is what we can expect for a log table), loose index scan can even degrade performance and generally will be avoided by MySQL.

We should use another approach here. Let’s create a sample table and test this approach:

Table creation details

CREATE TABLE filler (
        id INT NOT NULL PRIMARY KEY AUTO_INCREMENT
) ENGINE=Memory;

CREATE TABLE logfile (
        id INT NOT NULL PRIMARY KEY,
        sparse INT NOT NULL,
        dense INT NOT NULL,
        ts DATETIME NOT NULL,
        stuffing VARCHAR(100) NOT NULL,
        KEY ix_logfile_ts_id (ts, id),
        KEY ix_logfile_sparse_ts_id (sparse, ts, id),
        KEY ix_logfile_dense_ts_id (dense, ts, id)
) ENGINE=InnoDB;

DELIMITER $$

CREATE PROCEDURE prc_filler(cnt INT)
BEGIN
        DECLARE _cnt INT;
        SET _cnt = 1;
        WHILE _cnt <= cnt DO
                INSERT
                INTO    filler
                SELECT  _cnt;
                SET _cnt = _cnt + 1;
        END WHILE;
END
$$

DELIMITER ;

START TRANSACTION;
CALL prc_filler(500000);
COMMIT;

INSERT
INTO    logfile
SELECT  id,
        CEILING(RAND(20100824) * 30),
        CEILING(RAND(20100824 << 1) * 30000),
        '2010-08-24' - INTERVAL RAND(20100824 << 2) * 10000000 SECOND,
        LPAD('', 100, '*')
FROM    filler;

This table has 1,000,000 records.

Instead of a single field, person, I created two different fields: sparse and dense. The first one has 30 distinct values, while the second one has 30,000. This will help us to see how data distribution affects performance of different queries.

Let’s run our original queries. We’ll adjust them a little to help MySQL to pick correct plans:

SELECT  *
FROM    (
        SELECT  sparse, MAX(ts) AS last_update
        FROM    logfile
        GROUP BY
                sparse
        ) q
ORDER BY
        last_update DESC
LIMIT 20;

View query results

select `q`.`sparse` AS `sparse`,`q`.`last_update` AS `last_update` from (select `20100824_latest`.`logfile`.`sparse` AS `sparse`,max(`20100824_latest`.`logfile`.`ts`) AS `last_update` from `20100824_latest`.`logfile` group by `20100824_latest`.`logfile`.`sparse`) `q` order by `q`.`last_update` desc limit 20

SELECT  *
FROM    (
        SELECT  dense, MAX(ts) AS last_update
        FROM    logfile
        GROUP BY
                dense
        ) q
ORDER BY
        last_update DESC
LIMIT 20;

View query results

select `q`.`dense` AS `dense`,`q`.`last_update` AS `last_update` from (select `20100824_latest`.`logfile`.`dense` AS `dense`,max(`20100824_latest`.`logfile`.`ts`) AS `last_update` from `20100824_latest`.`logfile` group by `20100824_latest`.`logfile`.`dense`) `q` order by `q`.`last_update` desc limit 20

We see that both queries use the same plan and return 20 records, but the first one is instant, while the second one runs for 500 ms. Both queries use filesort, but in second case it has to sort 30,000 records (compared to 30 in the first case).

In this case, it is better to use another approach.

With our original query, we take each person and see which record is latest for this person. But we can as well do it the other way round: take the records in descending order, one by one, and for each record see if it’s latest for this person. If it is, we should return it; if it’s not, this means that the record for this person has already been returned (remember, we take them in descending order).

It’s easy to see that 20 records returned this way will, first, belong to 20 different people, and, second, be the latest records of their respective persons. This is exactly what we need.

The records can easily be scanned in the descending order using the index on (ts, id). But how do we check if the record is the latest? It’s simple: we just take the last record for the given person from the index on (person, ts, id) and compare its id. It takes but a single index seek per record and is almost instant.

Here’s the query to do it:

SELECT  id, sparse, dense, ts
FROM    logfile lf
WHERE   id =
        (
        SELECT  id
        FROM    logfile lfi
        WHERE   lfi.sparse = lf.sparse
        ORDER BY
                sparse DESC, ts DESC, id DESC
        LIMIT 1
        )
ORDER BY
        ts DESC, id DESC
LIMIT 20

Field or reference '20100824_latest.lf.sparse' of SELECT #2 was resolved in SELECT #1
select `20100824_latest`.`lf`.`id` AS `id`,`20100824_latest`.`lf`.`sparse` AS `sparse`,`20100824_latest`.`lf`.`dense` AS `dense`,`20100824_latest`.`lf`.`ts` AS `ts` from `20100824_latest`.`logfile` `lf` where (`20100824_latest`.`lf`.`id` = (select `20100824_latest`.`lfi`.`id` from `20100824_latest`.`logfile` `lfi` where (`20100824_latest`.`lfi`.`sparse` = `20100824_latest`.`lf`.`sparse`) order by `20100824_latest`.`lfi`.`sparse` desc,`20100824_latest`.`lfi`.`ts` desc,`20100824_latest`.`lfi`.`id` desc limit 1)) order by `20100824_latest`.`lf`.`ts` desc,`20100824_latest`.`lf`.`id` desc limit 20

As we can see, this query uses two different indexes. The first one on (ts, id) is used to scan all records according to the overall timeline; the second one, on (sparse, ts, id) is used to find the id of the latest entry for a person and check if it’s the same as the record selected from the general timeline.

The query is instant: 3 ms.

Let’s check the same query on a column with lots of values:

SELECT  id, sparse, dense, ts
FROM    logfile lf
WHERE   id =
        (
        SELECT  id
        FROM    logfile lfi
        WHERE   lfi.dense = lf.dense
        ORDER BY
                dense DESC, ts DESC, id DESC
        LIMIT 1
        )
ORDER BY
        ts DESC, id DESC
LIMIT 20

Field or reference '20100824_latest.lf.dense' of SELECT #2 was resolved in SELECT #1
select `20100824_latest`.`lf`.`id` AS `id`,`20100824_latest`.`lf`.`sparse` AS `sparse`,`20100824_latest`.`lf`.`dense` AS `dense`,`20100824_latest`.`lf`.`ts` AS `ts` from `20100824_latest`.`logfile` `lf` where (`20100824_latest`.`lf`.`id` = (select `20100824_latest`.`lfi`.`id` from `20100824_latest`.`logfile` `lfi` where (`20100824_latest`.`lfi`.`dense` = `20100824_latest`.`lf`.`dense`) order by `20100824_latest`.`lfi`.`dense` desc,`20100824_latest`.`lfi`.`ts` desc,`20100824_latest`.`lfi`.`id` desc limit 1)) order by `20100824_latest`.`lf`.`ts` desc,`20100824_latest`.`lf`.`id` desc limit 20

We see that the query is instant again, despite the data distribution being completely different. This is because the query only skips the records which are not the latest of their persons, and the total number of the records to scan is defined by how many records do we browse before we encounter the 20th unique value in our scan. This value decreases exponentially as the number of distinct persons in the table grows, but with 99% probability it won’t exceed 100 records even for only 20 distinct persons in the table.

The only problem that can arise here is that the number of distinct persons in the table is less than the LIMIT we set. In this case, no new records after the limit is reached can be returned, and a full index scan (accompanied by an index seek once per record) will ultimately be performed.

To work around this, the following simple query can be run in advance:

SELECT  COUNT(*)
FROM    (
        SELECT  DISTINCT sparse
        FROM    logfile
        LIMIT 20
        ) q

View query details

select count(0) AS `COUNT(*)` from (select distinct `20100824_latest`.`logfile`.`sparse` AS `sparse` from `20100824_latest`.`logfile` limit 20) `q`

This query will return the actual number of distinct persons in the table if there are less than 20 (or 20 if these are more).

This query is instant even for the dense data:

SELECT  COUNT(*)
FROM    (
        SELECT  DISTINCT dense
        FROM    logfile
        LIMIT 20
        ) q

View query details

select count(0) AS `COUNT(*)` from (select distinct `20100824_latest`.`logfile`.`dense` AS `dense` from `20100824_latest`.`logfile` limit 20) `q`

This needs to be run as a separate query because MySQL does not allow using anything other than constants in the LIMIT clause. The result of this query should be substituted into the LIMIT clause on the client or in a dynamically composed query on the server.

Summary

To select a number of latest unique records from a table, one can use aggregate functions, however, this can decrease the query performance.

This can be done more efficiently by creating two different indexes on the table and checking the records taken from the general timeline against the end of the index on the person’s timeline.

To avoid performance degradation in marginal cases (when the total number of persons in the table is less than LIMIT), it is possible to make an additional check for the total number of distinct records and adjust the LIMIT clause if there are not enough records.

P. S. I decided to enable comments for the technical posts as well. You are welcome to comment.

Frode Underhill asks:

I have some applications that are logging to a MySQL database table.

The table is pretty standard on the form:

time BIGINT(20)	source TINYINT(4)	severity ENUM	text VARCHAR(255)

, where source identifies the system that generated the log entry.

There are very many entries in the table (>100 million), of which 99.9999% are debug or info messages.

I’m making an interface for browsing this log, which means I’ll be doing queries like

SELECT  *
FROM    log
WHERE   source = 2
        AND severity IN (1,2)
        AND time > 12345
ORDER BY
        time ASC
LIMIT 30

, if I want to find debug or info log entries from a certain point in time, or

SELECT  *
FROM    log
WHERE   source = 2
        AND severity IN (1,2)
        AND time < 12345
ORDER BY
        time DESC
LIMIT 30

for finding entries right before a certain time.

How would one go about indexing & querying such a table?

I thought I had it figured out (I pretty much just tried every different combination of columns in an index), but there’s always some set of parameters that results in a really slow query.

The problem is that you cannot use a single index both for filtering and ordering if you have a ranged condition (severity IN (1, 2) in this case).

Recently I wrote an article with a proposal to improve SQL optimizer to handle these conditions. If a range has low cardinality (this is, there are few values that con possibly satisfy the range), then the query could be improved by rewriting the range as a series of individual queries, each one using one of the values constituting the range in an equijoin:

• Things SQL needs: determining range cardinality

No optimizers can handle this condition automatically yet, so we’ll need to emulate it.

Since the severity field is defined as an enum with only 5 values possible, any range condition on this field can be satisfied by no more than 5 distinct values, thus making this table ideal for rewriting the query.

Let’s create a sample table:

Table creation details

CREATE TABLE filler (
        id INT NOT NULL PRIMARY KEY AUTO_INCREMENT
) ENGINE=Memory;

CREATE TABLE t_log (
        id INT NOT NULL,
        ts BIGINT NOT NULL,
        source TINYINT(4) NOT NULL,
        severity ENUM('DEBUG','INFO','WARNING','ERROR','FATAL') NOT NULL,
        tx VARCHAR(255)
) ENGINE=MyISAM;

DELIMITER $$

CREATE PROCEDURE prc_filler(cnt INT)
BEGIN
        DECLARE _cnt INT;
        SET _cnt = 1;
        WHILE _cnt <= cnt DO
                INSERT
                INTO    filler
                SELECT  _cnt;
                SET _cnt = _cnt + 1;
        END WHILE;
END
$$

DELIMITER ;

START TRANSACTION;
CALL prc_filler(3500);
COMMIT;

INSERT
INTO    t_log
SELECT  (f1.id - 1) * 3000 + f2.id,
        UNIX_TIMESTAMP('2010-06-29' - INTERVAL (f1.id - 1) * 3000 + f2.id SECOND),
        CEILING(RAND(20100629) * 10),
        5 - FLOOR(LOG10(CEILING(RAND(20100629 << 1) * 99999))),
        CONCAT('Message ', (f1.id - 1) * 3000 + f2.id)
FROM    filler f1
CROSS JOIN
        filler f2;

CREATE INDEX ix_log_source_ts ON t_log (source, ts);

CREATE INDEX ix_log_source_severity_ts ON t_log (source, severity, ts);

This MyISAM table has 12,250,000 records, with 10 random sources (distributed evenly) and 5 random severities (distributed logarithmically):

SELECT  severity, COUNT(*)
FROM    t_log
GROUP BY
        severity;

We also created two indexes (one on (source, ts), the other one on (source, severity, ts)).

Now, let’s try to run some queries as is:

SELECT  *
FROM    t_log
WHERE   source = 2
        AND severity IN (1, 2)
        AND ts <= 1277754000
ORDER BY
        source DESC, ts DESC
LIMIT 30

select `20100630_range`.`t_log`.`id` AS `id`,`20100630_range`.`t_log`.`ts` AS `ts`,`20100630_range`.`t_log`.`source` AS `source`,`20100630_range`.`t_log`.`severity` AS `severity`,`20100630_range`.`t_log`.`tx` AS `tx` from `20100630_range`.`t_log` where ((`20100630_range`.`t_log`.`source` = 2) and (`20100630_range`.`t_log`.`severity` in (1,2)) and (`20100630_range`.`t_log`.`ts` <= 1277754000)) order by `20100630_range`.`t_log`.`source` desc,`20100630_range`.`t_log`.`ts` desc limit 30

This is very fast. It uses the index which does not include severity: since the values 1 and 2 are very frequent, it’s much more efficient just to filter them out. The index preserves the order, that’s why there is no filesort in the plan.

SELECT  *
FROM    t_log
WHERE   source = 2
        AND severity IN (4, 5)
        AND ts <= 1277754000
ORDER BY
        source DESC, ts DESC
LIMIT 30

select `20100630_range`.`t_log`.`id` AS `id`,`20100630_range`.`t_log`.`ts` AS `ts`,`20100630_range`.`t_log`.`source` AS `source`,`20100630_range`.`t_log`.`severity` AS `severity`,`20100630_range`.`t_log`.`tx` AS `tx` from `20100630_range`.`t_log` where ((`20100630_range`.`t_log`.`source` = 2) and (`20100630_range`.`t_log`.`severity` in (4,5)) and (`20100630_range`.`t_log`.`ts` <= 1277754000)) order by `20100630_range`.`t_log`.`source` desc,`20100630_range`.`t_log`.`ts` desc limit 30

This is very fast too. The index which includes severity is used (along with the filesort of course, because the order cannot be preserved with multiple values of severity), but the total number of records evaluated is so small that the filesort is not much of a problem.

Now, let’s try to include 3 into the query above:

SELECT  *
FROM    t_log
WHERE   source = 2
        AND severity IN (3, 4)
        AND ts <= 1277754000
ORDER BY
        source DESC, ts DESC
LIMIT 30

select `20100630_range`.`t_log`.`id` AS `id`,`20100630_range`.`t_log`.`ts` AS `ts`,`20100630_range`.`t_log`.`source` AS `source`,`20100630_range`.`t_log`.`severity` AS `severity`,`20100630_range`.`t_log`.`tx` AS `tx` from `20100630_range`.`t_log` where ((`20100630_range`.`t_log`.`source` = 2) and (`20100630_range`.`t_log`.`severity` in (3,4)) and (`20100630_range`.`t_log`.`ts` <= 1277754000)) order by `20100630_range`.`t_log`.`source` desc,`20100630_range`.`t_log`.`ts` desc limit 30

Now, this runs for almost 250 ms. Why?

There are 110,557 records with severity = 'WARNING'. This is too many for a filesort but too few for using where (filtering the records with the index that preserves the order). There will be too many records that will need to be skipped.

To work around this, we could combine the queries using UNION ALL. Since the original query uses ORDER BY and LIMIT, we may put them into two separate queries (which will yield 60 records) and finally apply it to the end resultset (to get the 30 records that are guaranteed to be contained among these 60):

SELECT  *
FROM    (
        SELECT  *
        FROM    t_log
        WHERE   source = 2
                AND severity = 3
                AND ts <= 1277754000
        ORDER BY
                source DESC, ts DESC
        LIMIT 30
        ) q
UNION ALL
SELECT  *
FROM    (
        SELECT  *
        FROM    t_log
        WHERE   source = 2
                AND severity = 4
                AND ts <= 1277754000
        ORDER BY
                source DESC, ts DESC
        LIMIT 30
        ) q
ORDER BY
        ts DESC
LIMIT 30

select `q`.`id` AS `id`,`q`.`ts` AS `ts`,`q`.`source` AS `source`,`q`.`severity` AS `severity`,`q`.`tx` AS `tx` from (select `20100630_range`.`t_log`.`id` AS `id`,`20100630_range`.`t_log`.`ts` AS `ts`,`20100630_range`.`t_log`.`source` AS `source`,`20100630_range`.`t_log`.`severity` AS `severity`,`20100630_range`.`t_log`.`tx` AS `tx` from `20100630_range`.`t_log` where ((`20100630_range`.`t_log`.`source` = 2) and (`20100630_range`.`t_log`.`severity` = 3) and (`20100630_range`.`t_log`.`ts` <= 1277754000)) order by `20100630_range`.`t_log`.`source` desc,`20100630_range`.`t_log`.`ts` desc limit 30) `q` union all select `q`.`id` AS `id`,`q`.`ts` AS `ts`,`q`.`source` AS `source`,`q`.`severity` AS `severity`,`q`.`tx` AS `tx` from (select `20100630_range`.`t_log`.`id` AS `id`,`20100630_range`.`t_log`.`ts` AS `ts`,`20100630_range`.`t_log`.`source` AS `source`,`20100630_range`.`t_log`.`severity` AS `severity`,`20100630_range`.`t_log`.`tx` AS `tx` from `20100630_range`.`t_log` where ((`20100630_range`.`t_log`.`source` = 2) and (`20100630_range`.`t_log`.`severity` = 4) and (`20100630_range`.`t_log`.`ts` <= 1277754000)) order by `20100630_range`.`t_log`.`source` desc,`20100630_range`.`t_log`.`ts` desc limit 30) `q` order by `ts` desc limit 30

This is much faster.

However, this solution requires composing the query dynamically, depending on the number of the severities in the condition. Is it possible to make this all in one static query that will accept the parameters in the IN list?

We can do it by using the applying the solution using to retrieve greatest-n-per-group in MySQL.

To do this, we will just select the 30th timestamp of each severity and find all records with the higher timestamps.

This can be done using a join:

SELECT  *
FROM    (
        SELECT  l.*
        FROM    (
                SELECT  source,
                        severity,
                        (
                        SELECT  ts
                        FROM    t_log li
                        WHERE   li.source = ss.source
                                AND li.severity = ss.severity
                                AND ts <= 1277754000
                        ORDER BY
                                li.source DESC, li.severity DESC, li.ts DESC
                        LIMIT 29, 1
                        ) AS mts
                FROM    (
                        SELECT  DISTINCT source, severity
                        FROM    t_log
                        WHERE   source = 2
                                AND severity IN (3, 4)
                        ) ss
                ) s
        JOIN    t_log l
        ON      l.source >= s.source
                AND l.source <= s.source
                AND l.severity = s.severity
                AND l.ts >= s.mts
                AND l.ts <= 1277754000
        ) q
ORDER BY
        ts DESC
LIMIT 30;

Field or reference 'ss.source' of SELECT #4 was resolved in SELECT #3
Field or reference 'ss.severity' of SELECT #4 was resolved in SELECT #3
select `q`.`id` AS `id`,`q`.`ts` AS `ts`,`q`.`source` AS `source`,`q`.`severity` AS `severity`,`q`.`tx` AS `tx` from (select `20100630_range`.`l`.`id` AS `id`,`20100630_range`.`l`.`ts` AS `ts`,`20100630_range`.`l`.`source` AS `source`,`20100630_range`.`l`.`severity` AS `severity`,`20100630_range`.`l`.`tx` AS `tx` from (select `ss`.`source` AS `source`,`ss`.`severity` AS `severity`,(select `20100630_range`.`li`.`ts` from `20100630_range`.`t_log` `li` where ((`20100630_range`.`li`.`source` = `ss`.`source`) and (`20100630_range`.`li`.`severity` = `ss`.`severity`) and (`20100630_range`.`li`.`ts` <= 1277754000)) order by `20100630_range`.`li`.`source` desc,`20100630_range`.`li`.`severity` desc,`20100630_range`.`li`.`ts` desc limit 29,1) AS `mts` from (select distinct `20100630_range`.`t_log`.`source` AS `source`,`20100630_range`.`t_log`.`severity` AS `severity` from `20100630_range`.`t_log` where ((`20100630_range`.`t_log`.`source` = 2) and (`20100630_range`.`t_log`.`severity` in (3,4)))) `ss`) `s` join `20100630_range`.`t_log` `l` where ((`20100630_range`.`l`.`severity` = `s`.`severity`) and (`20100630_range`.`l`.`source` >= `s`.`source`) and (`20100630_range`.`l`.`source` <= `s`.`source`) and (`20100630_range`.`l`.`ts` >= `s`.`mts`) and (`20100630_range`.`l`.`ts` <= 1277754000))) `q` order by `q`.`ts` desc limit 30

All possible values of source and severity are selected using a loose scan (which is instant since there are few of them). Each pair of values is then used as a join condition. A single index range satisfies each pair of values, so each join iteration uses an index efficiently (actually, the access path is reevaluated for each iteration as shown by Range checked for each record (index map: 0x3).

The total number of records that would be returned by this query be there no LIMIT is 60 or maybe even more (in case of ties on ts). However, we don’t need to resolve the ties in the subqueries, since the final ORDER BY / LIMIT does this for us.

The query completes in 4 ms which is instant. More than that, it does not need to be rewritten to handle different combinations of values: they could be provided in a single IN clause.

Hope that helps.

I’m always glad to answer the questions regarding database queries.

Ask me a question

Just a quick reminder: an anti-join is an operation that returns all records from one table which share a value of a certain column with no records from another table.

In SQL, there are at least three methods to implement it:

LEFT JOIN / IS NULL

SELECT  o.*
FROM    outer o
LEFT JOIN
        inner i
ON      i.value = o.value
WHERE   i.value IS NULL

NOT IN

SELECT  o.*
FROM    outer o
WHERE   o.value NOT IN
        (
        SELECT  value
        FROM    inner
        )

NOT EXISTS

SELECT  o.*
FROM    outer o
WHERE   NOT EXISTS
        (
        SELECT  NULL
        FROM    inner i
        WHERE   i.value = o.value
        )

When inner.value is marked as NOT NULL, all these queries are semantically equivalent and with proper indexing have similarly optimized execution plans in MySQL.

Now, what if inner.value is not nullable and does contain some NULL values?

Let’s create some sample tables:

Table creation details

CREATE TABLE filler (
        id INT NOT NULL PRIMARY KEY AUTO_INCREMENT
) ENGINE=MyISAM;

CREATE TABLE t_inner (
        id INT NOT NULL PRIMARY KEY,
        val INT,
        stuffing VARCHAR(200) NOT NULL,
        KEY ix_inner_val (val)
) ENGINE=MyISAM DEFAULT CHARSET=utf8;

CREATE TABLE t_outer (
        id INT NOT NULL PRIMARY KEY,
        val INT,
        stuffing VARCHAR(200) NOT NULL,
        KEY ix_outer_val (val)
) ENGINE=MyISAM DEFAULT CHARSET=utf8;

DELIMITER $$

CREATE PROCEDURE prc_filler(cnt INT)
BEGIN
        DECLARE _cnt INT;
        SET _cnt = 1;
        WHILE _cnt <= cnt DO
                INSERT
                INTO    filler
                SELECT  _cnt;
                SET _cnt = _cnt + 1;
        END WHILE;
END
$$

DELIMITER ;

START TRANSACTION;
CALL prc_filler(1000000);
COMMIT;

INSERT
INTO    t_inner
SELECT  id,
        NULLIF(CEILING(RAND(20100527) * 100000), 100000),
        RPAD('', 200, '*')
FROM    filler;

INSERT
INTO    t_outer
SELECT  id,
        NULLIF(CEILING(RAND(20100527 << 1) * 100000), 100000),
        RPAD('', 200, '*')
FROM    filler;

There are two identical MyISAM tables. Each of the tables contains 1,000,000 random values from 1 to 99,999 and also some NULL values. There is an index on value in both tables.

Now, let’s check the queries.

NOT EXISTS

SELECT  SUM(LENGTH(stuffing)), COUNT(*)
FROM    t_outer o
WHERE   NOT EXISTS
        (
        SELECT  NULL
        FROM    t_inner i
        WHERE   i.val = o.val
        )

Field or reference '20100527_anti.o.val' of SELECT #2 was resolved in SELECT #1
select sum(length(`20100527_anti`.`o`.`stuffing`)) AS `SUM(LENGTH(stuffing))`,count(0) AS `COUNT(*)` from `20100527_anti`.`t_outer` `o` where (not(exists(select NULL from `20100527_anti`.`t_inner` `i` where (`20100527_anti`.`i`.`val` = `20100527_anti`.`o`.`val`))))

The query completes in 9.9 seconds. As we can see, it is optimized to use the index on t_inner.val and return on the first match.

LEFT JOIN / IS NULL

SELECT  SUM(LENGTH(o.stuffing)), COUNT(*)
FROM    t_outer o
LEFT JOIN
        t_inner i
ON      i.val = o.val
WHERE   i.id IS NULL

select sum(length(`20100527_anti`.`o`.`stuffing`)) AS `SUM(LENGTH(o.stuffing))`,count(0) AS `COUNT(*)` from `20100527_anti`.`t_outer` `o` left join `20100527_anti`.`t_inner` `i` on((`20100527_anti`.`i`.`val` = `20100527_anti`.`o`.`val`)) where isnull(`20100527_anti`.`i`.`id`)

The query semantics are the same as those of NOT EXISTS, and we even see the Not exists optimization in the plan, however this query performs much more poorly than NOT EXISTS: 13 seconds. Why?

MySQL documentation on EXPLAIN states that Not exists is used to optimize the queries similar to the one we have just run: LEFT JOIN with IS NULL predicate applied to a non-nullable column.

MySQL is aware that such a predicate can only be satisfied by a record resulting from a JOIN miss (i. e. when no matching record was found in the rightmost table) and stops reading records after first index hit.

However, this optimization is implemented in a way that is far from being perfect. Despite the fact that no actual value of id can be returned by such a query, the engine still looks up id in the table (since it’s not a part of the index). We can see it in the plan: unlike NOT EXISTS query, there is no Using index for t_inner. This means that a table lookup is performed.

Even we replace id with val in the query, it still performs poorly:

SELECT  SUM(LENGTH(o.stuffing)), COUNT(*)
FROM    t_outer o
LEFT JOIN
        t_inner i
ON      i.val = o.val
WHERE   i.val IS NULL

select sum(length(`20100527_anti`.`o`.`stuffing`)) AS `SUM(LENGTH(o.stuffing))`,count(0) AS `COUNT(*)` from `20100527_anti`.`t_outer` `o` left join `20100527_anti`.`t_inner` `i` on((`20100527_anti`.`i`.`val` = `20100527_anti`.`o`.`val`)) where isnull(`20100527_anti`.`i`.`val`)

This time, no table lookups are made but there is no Not exists optimization either.

Despite the fact that the join condition eliminates possibility of an actual NULL being returned by the query and any val IS NULL reaching the WHERE clause is a result of a join miss, MySQL still examines all records in t_inner, not stopping after the first hit.

This had been submitted as a bug.

Now, what about NOT IN?

NOT IN

Unlike the previous two queries that only differ in implementation, not in semantics, NOT IN, being applied as is, would yield the different results.

NOT EXISTS and IS NULL are two-state predicates, they can only return TRUE or FALSE. NOT IN is a three-state predicate: it can return TRUE, FALSE or NULL.

NULL value is returned in two cases:

• When t_outer.value being tested is NULL
• When at least one of t_inner.value is NULL

This means that having but a single NULL in t_inner would prevent the query from returning anything.

Naive approach

Let’s see what happens if we just substitute NOT IN instead of NOT EXISTS:

SELECT  SUM(LENGTH(stuffing)), COUNT(*)
FROM    t_outer o
WHERE   val NOT IN
        (
        SELECT  val
        FROM    t_inner i
        )

select sum(length(`20100527_anti`.`o`.`stuffing`)) AS `SUM(LENGTH(stuffing))`,count(0) AS `COUNT(*)` from `20100527_anti`.`t_outer` `o` where (not(<in_optimizer>(`20100527_anti`.`o`.`val`,<exists>(<index_lookup>(<cache>(`20100527_anti`.`o`.`val`) in t_inner on ix_inner_val checking NULL having trigcond(<is_not_null_test>(`20100527_anti`.`i`.`val`)))))))

Since there are NULLs in t_inner, no record in t_outer can satisfy the predicate.

MySQL does not optimize this very well. It takes but a single index scan to find out if there are NULL values in t_inner and return if they are, but for some reason MySQL still applies the condition to each record in t_outer.

Naive approach, improved

With a little help from our side, this can be improved:

SELECT  SUM(LENGTH(stuffing)), COUNT(*)
FROM    t_outer o
WHERE   NOT EXISTS
        (
        SELECT  NULL
        FROM    t_inner i
        WHERE   val IS NULL
        )
        AND val NOT IN
        (
        SELECT  val
        FROM    t_inner i
        )

select sum(length(`20100527_anti`.`o`.`stuffing`)) AS `SUM(LENGTH(stuffing))`,count(0) AS `COUNT(*)` from `20100527_anti`.`t_outer` `o` where 0

We added an explicit check for NULL values. Since it’s not correlated, MySQL could instantly prove it false, cache it and avoid the table scan at all.

Ignoring right side NULLs

Now, let’s make a NOT IN query that does not take the NULL values in t_inner into account:

SELECT  SUM(LENGTH(stuffing)), COUNT(*)
FROM    t_outer o
WHERE   val NOT IN
        (
        SELECT  val
        FROM    t_inner i
        WHERE   val IS NOT NULL
        )

select sum(length(`20100527_anti`.`o`.`stuffing`)) AS `SUM(LENGTH(stuffing))`,count(0) AS `COUNT(*)` from `20100527_anti`.`t_outer` `o` where (not(<in_optimizer>(`20100527_anti`.`o`.`val`,<exists>(<index_lookup>(<cache>(`20100527_anti`.`o`.`val`) in t_inner on ix_inner_val checking NULL where (`20100527_anti`.`i`.`val` is not null) having trigcond(<is_not_null_test>(`20100527_anti`.`i`.`val`)))))))

This time, the query returns records, but not as many as the previous queries did.

We made an additional check for NULL in t_inner but not in t_outer. There are some records in t_outer that have a NULL in val. Both IN and NOT IN would evaluate to NULL and WHERE would filter them out.

We see another glitch in MySQL optimizer here: a Full scan on NULL key applied. Since NOT IN should always return TRUE when the subquery returns no records (even if the value checked is a NULL), on correlated queries a fullscan should be applied to check for the records and find out whether to return NULL or FALSE. However, in this case the IN subquery is not correlated, so the check could only be performed once and cached, like with the LEFT JOIN.

In our case the overhead would be negligible, since the subquery would return on the first match, but it could matter if we had more NULL values in t_outer.

Now, what if we want NULL records on t_outer to be returned as well? We just need to add an additional check for NULLs.

Ignoring all NULLs

SELECT  SUM(LENGTH(stuffing)), COUNT(*)
FROM    t_outer o
WHERE   val IS NULL
        OR val NOT IN
        (
        SELECT  val
        FROM    t_inner i
        WHERE   val IS NOT NULL
        )

select sum(length(`20100527_anti`.`o`.`stuffing`)) AS `SUM(LENGTH(stuffing))`,count(0) AS `COUNT(*)` from `20100527_anti`.`t_outer` `o` where (isnull(`20100527_anti`.`o`.`val`) or (not(<in_optimizer>(`20100527_anti`.`o`.`val`,<exists>(<index_lookup>(<cache>(`20100527_anti`.`o`.`val`) in t_inner on ix_inner_val checking NULL where (`20100527_anti`.`i`.`val` is not null) having trigcond(<is_not_null_test>(`20100527_anti`.`i`.`val`))))))))

Here, the query returns the same results as NOT EXISTS.

Full scan on NULL key is still present in the plan but will never actually be executed because it will be short circuited by the previous IS NULL check.

Summary

As was shown in the earlier article, LEFT JOIN / IS NULL and NOT IN are best used to implement an anti-join in MySQL if the columns on both sides are not nullable.

The situation is different when the columns are nullable:

• NOT EXISTS performs in most straightforward way: just checks equality and returns TRUE or FALSE on the first hit / miss.
• LEFT JOIN / IS NULL either makes an additional table lookup or does not return on the first match and performs more poorly in both cases.
• NOT IN, having different semantics, requires additional checks for NULL values. These checks should be coded into the query

With nullable columns, NOT EXISTS and NOT IN (with additional checks for NULLS) are the most efficient methods to implement an anti-join in MySQL.

LEFT JOIN / IS NULL performs poorly.

Ivo Radev asks:

I am trying to make a very simple query.

We have a log table which different machines write to. Given the machine list, I need to find the latest log timestamp.

Currently, the query looks like this:

SELECT  MAX(log_time)
FROM    log_table
WHERE   log_machine IN ($machines)

, and I pass the comma-separated list of $machines from PHP.

The weird thing is that the query is literally instant when there is only one machine (any) in the list but slow when there are multiple machines.

I’m considering doing it in separate queries and then process the results in PHP. However I’d like to know if there is a fast solution in MySQL.

Most probably, there is a composite index on (log_machine, log_time) which is being used for the query.

Usually, a query like this:

SELECT  MAX(log_time)
FROM    log_table

on the indexed field log_time can be served with a single index seek on the index.

Indeed, the MAX(log_time), by definition, is the latest entry in the index order, and can be fetched merely by finding the trailing index entry. It’s a matter of several page reads in the B-Tree, each one following the rightmost link to the lower-level page.

Similarly, this query:

SELECT  MAX(log_time)
FROM    log_table
WHERE   log_machine = $my_machine

can be served with a single index seek too. However, the index should include log_machine as a leading column.

In this case, a set of records satisfying the WHERE clause of the query is represented by a single logically continuous block of records in the index, each one sharing the same value of log_machine. MAX(log_time) will of course be held by the last record in this block. MySQL just finds that last record and takes the log_time out of it.

Now, what if we have a multiple condition on log_machine?

The index remains the same, but the record holding MAX(log_time) is not the last record in a single continuous block anymore. Instead, there are multiple blocks each having its own MAX(log_date). log_time cannot be found merely by taking the last record from the index block: it is not known which one is the correct one.

On composite indexes, however, MySQL offers loose index scan. This means that it jumps over the distinct values of the leading column, doing an index seek (instead of index scan) to retrieve each next value.

As stated in the documentation, this method is ideal to doing the queries like that:

SELECT  log_machine, MAX(log_time)
FROM    log_table
WHERE   log_machine IN ($my_machine_list)

As we said earlier, for each log_machine, its MAX(log_time) can be returned very fast, and the list of the log_machines could be obtained with a loose index scan, by seeking the keys in the index.

This query, however, will not produce a single MAX(log_time): instead, it will return as many maximums as there are values in the list (which are found in the table, of course).

But this can be easily worked around: we just select the greatest one of these records. Since the subquery will only return several records, the greatest one if them can be found almost instantly.

Let’s create a sample table:

Table creation details

CREATE TABLE filler (
        id INT NOT NULL PRIMARY KEY AUTO_INCREMENT
) ENGINE=Memory;

CREATE TABLE log_table (
        id INT NOT NULL PRIMARY KEY,
        log_machine VARCHAR(20) NOT NULL,
        log_time DATETIME NOT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

DELIMITER $$

CREATE PROCEDURE prc_filler(cnt INT)
BEGIN
        DECLARE _cnt INT;
        SET _cnt = 1;
        WHILE _cnt <= cnt DO
                INSERT
                INTO    filler
                SELECT  _cnt;
                SET _cnt = _cnt + 1;
        END WHILE;
END
$$

DELIMITER ;

START TRANSACTION;
CALL prc_filler(1000000);
COMMIT;

INSERT
INTO    log_table
SELECT  id,
        CONCAT('Machine ' , CEILING(RAND(20100508) * 10)),
        '2010-05-08' - INTERVAL CEILING(RAND(20100508 << 1) * 10000000) SECOND
FROM    filler f1;

CREATE INDEX ix_log_machine_time ON log_table (log_machine, log_time);

The table has 1,000,000 records

Now, let’s see how the original query performs:

SELECT  MAX(log_time) AS maxtime
FROM    log_table
WHERE   log_machine IN ('Machine 3', 'Machine 5', 'Machine 7', 'Machine 9')

select max(`20100508_max`.`log_table`.`log_time`) AS `maxtime` from `20100508_max`.`log_table` where (`20100508_max`.`log_table`.`log_machine` in ('Machine 3','Machine 5','Machine 7','Machine 9'))

The query uses range access to retrieve the records and browses all records to find the maximum. It takes 640 ms on a table 1,000,000 log records (which is about a day’s output of a single web server under a load decent but not super hard).

Now, let’s try to select the greatest of the group-wise maximums:

SELECT  MAX(log_time) AS maxtime
FROM    log_table
WHERE   log_machine IN ('Machine 3', 'Machine 5', 'Machine 7', 'Machine 9')
GROUP BY
        log_machine
ORDER BY
        1 DESC
LIMIT 1

select max(`20100508_max`.`log_table`.`log_time`) AS `maxtime` from `20100508_max`.`log_table` where (`20100508_max`.`log_table`.`log_machine` in ('Machine 3','Machine 5','Machine 7','Machine 9')) group by `20100508_max`.`log_table`.`log_machine` order by 1 desc limit 1

Now, it’s instant, as it should be.

As it often happens, by appending three seemingly redundant clauses to a query we made MySQL to choose a more efficient plan and the query is now instant even with multiple machines in the list.

Hope that helps.

I’m always glad to answer the questions regarding database queries.

Ask me a question

Nate asks:

I know you’ve addressed similar issues related to the greatest-per-group query but this seems to be a different take on that.

Example table:

t_group

item_id	group_id	score
100	1	2
100	2	3
200	1	1
300	1	4
300	2	2

Each item may be in multiple groups. Each instance of an item in that group is given a score (how relevant it is the the group).

So given the data above, when querying for group 1 it should return items 200 and 300 (item 100‘s highest score is for group 2, so it’s excluded).

The classical greatest-n-per-group problem requires selecting a single record from each group holding a group-wise maximum. This case is a little bit different: for a given group, we need to select all records holding an item-wise maximum.

Let’s create a sample table:

Table creation details

CREATE TABLE filler (
        id INT NOT NULL PRIMARY KEY AUTO_INCREMENT
) ENGINE=Memory;

CREATE TABLE t_groups (
        item_id INT NOT NULL,
        group_id INT NOT NULL,
        score INT NOT NULL,
        PRIMARY KEY (group_id, item_id),
        KEY ix_groups_gsi (item_id, score, group_id)
) ENGINE=InnoDB;              

DELIMITER $$

CREATE PROCEDURE prc_filler(cnt INT)
BEGIN
        DECLARE _cnt INT;
        SET _cnt = 1;
        WHILE _cnt <= cnt DO
                INSERT
                INTO    filler
                SELECT  _cnt;
                SET _cnt = _cnt + 1;
        END WHILE;
END
$$

DELIMITER ;

START TRANSACTION;
CALL prc_filler(1000000);
COMMIT;

INSERT
INTO    t_groups
SELECT  (id - 1) % 1000 + 1,
        (id - 1) div 1000 + 1,
        CEILING(RAND(20100422) * 10000)
FROM    filler;

This table contains 1,000,000 records: 1,000 items in 1,000 groups with random scores.

Let’s write a query which would return us all items whose largest score is in group 1.

To do this, we need to select all items from group 1 and check that no other group has a greater value of score for that item. The most intuitive query for this would look like that:

SELECT  *
FROM    t_groups go
WHERE   group_id = 1
        AND NOT EXISTS
        (
        SELECT  group_id
        FROM    t_groups gi
        WHERE   gi.item_id = go.item_id
                AND (gi.score, gi.group_id) > (go.score, go.group_id)
        )

Field or reference '20100422_rank.go.item_id' of SELECT #2 was resolved in SELECT #1
Field or reference '20100422_rank.go.score' of SELECT #2 was resolved in SELECT #1
Field or reference '20100422_rank.go.group_id' of SELECT #2 was resolved in SELECT #1
select `20100422_rank`.`go`.`item_id` AS `item_id`,`20100422_rank`.`go`.`group_id` AS `group_id`,`20100422_rank`.`go`.`score` AS `score` from `20100422_rank`.`t_groups` `go` where ((`20100422_rank`.`go`.`group_id` = 1) and (not(exists(select `20100422_rank`.`gi`.`group_id` AS `group_id` from `20100422_rank`.`t_groups` `gi` where ((`20100422_rank`.`gi`.`item_id` = `20100422_rank`.`go`.`item_id`) and ((`20100422_rank`.`gi`.`score`,`20100422_rank`.`gi`.`group_id`) > (`20100422_rank`.`go`.`score`,`20100422_rank`.`go`.`group_id`)))))))

This query returns us 3 items and their score. We see that these items are not ranked higher in any other group.

However, this query is quite inefficient: it takes almost half a second. This is way too long for 1K items. Let’s look how can it be optimized.

If we look into the query plan we will see that no range access path is used for the subquery, despite the condition that can easily be optimized used such an access path. Instead, MySQL uses only the ref access on item_id, combined with Using where; Using index.

What does it all mean?

We have the composite index on (item_id, score, group_id). This means that within the index, the records are ordered first on item_id then by score and then, in a case of a tie, by group_id.

For group 1, MySQL should perform 1,000 comparisons: for each item within the group, the engine should make sure that no other group has a higher score for the same item.

Ideally, MySQL should have found each given set of (item_id, score, group_id) in the index and then just make a single next key search to check if this record is last in the index within the given item_id. That would show as a range search in the query plan, since we actually are checking values between (item_id, score, group_id) and (item_id, +INF, +INF).

However, MySQL cannot do such things in a correlated subquery. Instead, it uses the ref access path: takes all records with the given item_id (i. e. between (item_id, -INF, -INF) and (item_id, +INF, +INF)) and traverses them applying the WHERE filter to each record.

For group 1, only 3 items are returned. This means that for majority of items (997 items), the whole index range for the items (1,000 records per item) had to be scanned.

And since the matching record is always the last one in the index range (because it’s the one with the greatest score), this means that even for returned items, the whole ranged had to be scanned too. The only difference is that the last record in the range satisfies the WHERE condition. No wonder it takes so long.

To speed up the query we need to trick MySQL a little.

Since it’s always the last record in the index range that satisfies the WHERE condition, why don’t we just take it and compare? If we see that it holds our group, we return TRUE, if we don’t, we return FALSE right away.

Here’s how we can do this:

SELECT  *
FROM    t_groups go
WHERE   group_id = 1
        AND group_id =
        (
        SELECT  group_id
        FROM    t_groups gi
        WHERE   gi.item_id = go.item_id
        ORDER BY
                item_id DESC, score DESC, group_id DESC
        LIMIT 1
        )

Field or reference '20100422_rank.go.item_id' of SELECT #2 was resolved in SELECT #1
select `20100422_rank`.`go`.`item_id` AS `item_id`,`20100422_rank`.`go`.`group_id` AS `group_id`,`20100422_rank`.`go`.`score` AS `score` from `20100422_rank`.`t_groups` `go` where ((`20100422_rank`.`go`.`group_id` = 1) and (1 = (select `20100422_rank`.`gi`.`group_id` AS `group_id` from `20100422_rank`.`t_groups` `gi` where (`20100422_rank`.`gi`.`item_id` = `20100422_rank`.`go`.`item_id`) order by `20100422_rank`.`gi`.`item_id` desc,`20100422_rank`.`gi`.`score` desc,`20100422_rank`.`gi`.`group_id` desc limit 1)))

Instead of checking the existence of the last row in a range, we check it’s value. This means that exactly one index record will be evaluated for each item within the group.

Note that the execution plan used by this query looks exactly the same as the first query’s one. Same ref condition on item_id, same Using where; Using index. The only difference (not shown in the EXPLAIN output) it that the index is traversed in descending order now. Instead of fetching 1,000 records from the beginning to check the existence of the last one, we just take one record from the end and check its value.

Those who read my blog already know why do we use seemingly redundant item_id DESC in the ORDER BY clause here. To make MySQL to use descending index access path instead of a filesort in an ordered query, we should list all clauses that constitute the index, even if some of them are filtered by the WHERE condition.

The second query completes in only 10 ms which is 40 times as fast as the original query.

Hope that helps.

I’m always glad to answer the questions regarding database queries.

Ask me a question

James asks:

Your series on hierarchical queries in MySQL is tremendous! I’m using it to create a series of threaded conversations.

I’m wondering if there is a way to paginate these results.

Specifically, let’s say I want to limit the conversations to return 10 root nodes (parent=0) and all of their children in a query.

I can’t just limit the final query, because that will clip off children. I’ve tried to add LIMITs to your stored functions, but I’m not getting the magic just right.

How would you go about doing this?

A quick reminder: MySQL does not support recursion (either CONNECT BY style or recursive CTE style), so using an adjacency list model is a somewhat complicated task.

However, it is still possible. The main idea is storing the recursion state in a session variable and call a user-defined function repeatedly to iterate over the tree, thus emulating recursion. The article mentioned in the question shows how to do that.

Normally, reading and assigning session variables in the same query is discouraged in MySQL, since the order of evaluation is not guaranteed. However, in the case we only use the table as a dummy recordset and no values of the records are actually used in the function, so the actual values returned by the function are completely defined by the function itself. The table is only used to ensure that the function is called enough times, and to present its results in form of a native resultset (which can be returned or joined with).

To do something with the logic of the function (like, imposing a limit on the parent nodes without doing the same on the child nodes), we, therefore, should tweak the function code, not the query that calls the functions. The only thing that matters in such a query is the number of records returned and we don’t know it in design time.

Limiting the parent nodes is quite simple: we just use another session variable to track the number of parent branches yet to be returned and stop processing as soon as the limit is hit, that is the variable becomes zero.

Let’s create a sample table and see how to do this:

Table creation details

CREATE TABLE filler (
        id INT NOT NULL PRIMARY KEY AUTO_INCREMENT
) ENGINE=Memory;

CREATE TABLE t_hierarchy (
        id int(10) unsigned NOT NULL AUTO_INCREMENT,
        parent int(10) unsigned NOT NULL,
        root INT NOT NULL,
        PRIMARY KEY (id),
        KEY ix_hierarchy_parent (parent, id),
        KEY ix_hierarchy_root (root)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

DELIMITER $$

CREATE PROCEDURE prc_filler(cnt INT)
BEGIN
        DECLARE _cnt INT;
        SET _cnt = 1;
        WHILE _cnt <= cnt DO
                INSERT
                INTO    filler
                SELECT  _cnt;
                SET _cnt = _cnt + 1;
        END WHILE;
END
$$

DELIMITER ;

START TRANSACTION;
CALL prc_filler(100000);
COMMIT;

INSERT
INTO    t_hierarchy
SELECT  id,
        CASE (id - 1) % 8
        WHEN 0 THEN 0
        ELSE FLOOR((((id - 1) % 8) + 1) / 2) + ((id - 1) div 8) * 8
        END,
        ((id - 1) div 8) * 8 + 1
FROM    filler;

There are 100,000 hierarchical records in multiple trees (8 records in each tree).

To limit the number of trees returned, we create the function similar to the one created in the earlier posts and add a little condition that would decrease the session variable, @parent_limit, each time a parent entry is returned. When this variable hits zero, it’s a signal to stop processing the records:

CREATE FUNCTION hierarchy_connect_by_parent_eq_prior_id(value INT) RETURNS INT
NOT DETERMINISTIC
READS SQL DATA
BEGIN
        DECLARE _id INT;
        DECLARE _parent INT;
        DECLARE _next INT;
        DECLARE CONTINUE HANDLER FOR NOT FOUND SET @id = NULL;

        SET _parent = @id;
        SET _id = -1;

        IF @id IS NULL THEN
                RETURN NULL;
        END IF;

        LOOP
                SELECT  MIN(id)
                INTO    @id
                FROM    t_hierarchy
                WHERE   parent = _parent
                        AND id > _id;
                IF @id IS NOT NULL OR _parent = @start_with THEN
                        SET @level = @level + 1;
                        IF _parent = @start_with AND @parent_limit > 0 THEN
                                SET @parent_limit = @parent_limit - 1;
                        END IF;
                        IF @parent_limit = 0 THEN
                                SET @id = NULL;
                        END IF;
                        RETURN @id;
                END IF;
                SET @level := @level - 1;
                SELECT  id, parent
                INTO    _id, _parent
                FROM    t_hierarchy
                WHERE   id = _parent;
        END LOOP;
END;

Let’s check it:

SELECT  CONCAT(REPEAT('    ', level - 1), CAST(hi.id AS CHAR)) AS treeitem, parent, level, lmt
FROM    (
        SELECT  hierarchy_connect_by_parent_eq_prior_id(id) AS id, @level AS level, @parent_limit as lmt
        FROM    (
                SELECT  @start_with := 0,
                        @parent_limit := 4,
                        @id := @start_with,
                        @level := 0
                ) vars, t_hierarchy
        WHERE   @id IS NOT NULL
        ) ho
JOIN    t_hierarchy hi
ON      hi.id = ho.id

select concat(repeat('    ',(`ho`.`level` - 1)),cast(`20100418_limit`.`hi`.`id` as char charset latin1)) AS `treeitem`,`20100418_limit`.`hi`.`parent` AS `parent`,`ho`.`level` AS `level`,`ho`.`lmt` AS `lmt` from (select `hierarchy_connect_by_parent_eq_prior_id`(`20100418_limit`.`t_hierarchy`.`id`) AS `id`,(@level) AS `level`,(@parent_limit) AS `lmt` from (select (@start_with:=0) AS `@start_with := 0`,(@parent_limit:=4) AS `@parent_limit := 4`,(@id:=(@start_with)) AS `@id := @start_with`,(@level:=0) AS `@level := 0`) `vars` join `20100418_limit`.`t_hierarchy` where ((@id) is not null)) `ho` join `20100418_limit`.`t_hierarchy` `hi` where (`20100418_limit`.`hi`.`id` = `ho`.`id`)

The first 3 branches in a proper hierarchy, almost instantly. Note that we need set @parent_limit to 4, i. e. one greater than the value we need.

Note that it is possible to achieve the similar behavior without using any recursive functions at all.

On most forums the filtering is performed on the topic starters, so it’s often a good idea to store the id of the topic starter along with each reply. In the sample table a did that as well.

With this, the filtering required becomes very simple:

SELECT  h.*
FROM    (
        SELECT  id
        FROM    t_hierarchy
        WHERE   parent = 0
        ORDER BY
                id
        LIMIT 3
        ) q
JOIN    t_hierarchy h
ON      h.root = q.id

select `20100418_limit`.`h`.`id` AS `id`,`20100418_limit`.`h`.`parent` AS `parent`,`20100418_limit`.`h`.`root` AS `root` from (select `20100418_limit`.`t_hierarchy`.`id` AS `id` from `20100418_limit`.`t_hierarchy` where (`20100418_limit`.`t_hierarchy`.`parent` = 0) order by `20100418_limit`.`t_hierarchy`.`id` limit 3) `q` join `20100418_limit`.`t_hierarchy` `h` where (`20100418_limit`.`h`.`root` = `q`.`id`)

This solution (that had been used on numerous forum engines for ages) is more efficient, since no function calls are involved, and more simple too.

However, it does not preserve the hierarchical order and does not allow sorting on anything but the topic starter, so if you need anything of these, the recursive function is still a way to go.

Hope that helps.

I’m always glad to answer the questions regarding database queries.

Ask me a question

Andrew Stillard asks:

I have a store which will hold around 50,000 products in a products table. Each product will have 14 options, giving 700,000 options in total. These are held in an options table which is joined via the product id.

Users search for products based on the options via an Advanced Search menu.

The users need to be able to select multiple options upon which to query. I would normally use a JOIN if it was just the one option to select upon, but because its a variable number i thought it would be best to loop through the WHERE EXISTS statement.

The issue i have currently is that the query is taking a minimum of 18 seconds (And that was a query when the tables only had a fraction of the total products in).
If you can help us speed this up, or suggest an alternative idea that would be greatly appreciated.

The option table mentioned here is in fact an implementation of the EAV model in a relational database.

Each record basically contains 3 things: id of the product it describes; id of the option and the value of the given option for the given product. These fields represent entity, attribute and value, respectively.

This model is very easy to maintain and expand should the need arise: all we have to do to define an extra attribute is to add a record to the EAV table with the name and the value of the attribute. This is a DML operation rather than a DDL one.

However, this model has a serious drawback: we cannot efficiently search for two or more options at once. An index can only be defined on several fields from a single record, so we can only search for a single option using an index.

There are two approaches to writing a query which would search for the entities with the certain conditions on several attributes at once:

1. For each attribute, find all entities for which the conditions on the given attribute hold, then aggregate the resulting entities, using COUNT(*) as a filter. The number of entity entries should be equal to the number of the attributes.
2. Takes each entity and for each attribute, check if the condition holds.

The first approach uses a GROUP BY, the second one uses EXISTS.

Let’s create a sample table and see which one is more efficient:

Table creation details

CREATE TABLE filler (
        id INT NOT NULL PRIMARY KEY AUTO_INCREMENT
) ENGINE=Memory;

CREATE TABLE t_product (
        id INT NOT NULL PRIMARY KEY,
        name VARCHAR(30) NOT NULL
) ENGINE=InnoDB DEFAULT CHARSET=UTF8;

CREATE TABLE t_option (
        product_id INT NOT NULL,
        option_id INT NOT NULL,
        value INT NOT NULL,
        PRIMARY KEY pk_option_o_p (product_id, option_id),
        KEY ix_option_o_v (option_id, value)
) ENGINE=InnoDB DEFAULT CHARSET=UTF8;

DELIMITER $$

CREATE PROCEDURE prc_filler(cnt INT)
BEGIN
        DECLARE _cnt INT;
        SET _cnt = 1;
        WHILE _cnt <= cnt DO
                INSERT
                INTO    filler
                SELECT  _cnt;
                SET _cnt = _cnt + 1;
        END WHILE;
END
$$

DELIMITER ;

START TRANSACTION;
CALL prc_filler(500000);
COMMIT;

INSERT
INTO    t_product
SELECT  id, CONCAT('Product ', id)
FROM    filler
ORDER BY
        id
LIMIT 50000;

INSERT
INTO    t_option
SELECT  p.id, f.id, CEILING(RAND(20100402 << 1) * 100)
FROM    t_product p
CROSS JOIN
        (
        SELECT  id
        FROM    filler
        ORDER BY
                id
        LIMIT 40
        ) f;

Table t_product contains 50,000 products; table t_option (an EAV table) contains the values of 40 attributes for each products, randomly filled.

Assume we got a complex query involving attributes from 1 to 6. The value of attribute 1 should be from 0 to 100; that of attribute 2 should be from 10 to 90, etc., finally, the value of the attribute 6 should be from 50 to 50.

GROUP BY

Using the first approach, we should find all product_id‘s in the EAV table (t_option) that satisfy the ranges for each attribute, then aggregate these products and filter those whose COUNT(*) is 6. This will mean that the product was found in all 6 ranges and hence satisfies all conditions.

Here’s the query to do that:

SELECT  COUNT(*)
FROM    (
        SELECT  product_id
        FROM    (
                SELECT  product_id
                FROM    (
                        SELECT  1 AS opt, 0 AS l, 100 AS h
                        UNION ALL
                        SELECT  2 AS opt, 10 AS l, 90 AS h
                        UNION ALL
                        SELECT  3 AS opt, 20 AS l, 80 AS h
                        UNION ALL
                        SELECT  4 AS opt, 30 AS l, 70 AS h
                        UNION ALL
                        SELECT  5 AS opt, 40 AS l, 60 AS h
                        UNION ALL
                        SELECT  6 AS opt, 50 AS l, 50 AS h
                        ) v
                JOIN    t_option o
                ON      o.option_id >= opt
                        AND o.option_id <= opt
                        AND o.value BETWEEN l AND h
                ) o
        GROUP BY
                o.product_id
        HAVING  COUNT(*) = 6
        ) q;

select count(0) AS `COUNT(*)` from (select `o`.`product_id` AS `product_id` from (select `20100402_options`.`o`.`product_id` AS `product_id` from (select 1 AS `opt`,0 AS `l`,100 AS `h` union all select 2 AS `opt`,10 AS `l`,90 AS `h` union all select 3 AS `opt`,20 AS `l`,80 AS `h` union all select 4 AS `opt`,30 AS `l`,70 AS `h` union all select 5 AS `opt`,40 AS `l`,60 AS `h` union all select 6 AS `opt`,50 AS `l`,50 AS `h`) `v` join `20100402_options`.`t_option` `o` where ((`20100402_options`.`o`.`option_id` >= `v`.`opt`) and (`20100402_options`.`o`.`option_id` <= `v`.`opt`) and (`20100402_options`.`o`.`value` between `v`.`l` and `v`.`h`))) `o` group by `o`.`product_id` having (count(0) = 6)) `q`

Note that there are two quite counterintuitive things in the query:

• o.option_id >= opt AND o.option_id <= opt instead of mere o.option_id = opt
• GROUP BY is applied to a nested query rather than being put immediately after the WHERE clause

As many readers of my blog will remember, these tricks are intended to make MySQL use Range checked for each record that we can spot in the plan. If not for these tricks, MySQL would use an index scan on only a part of the composite index on (attribute, value, product), since the joins on the mixed equality/range conditions are not optimized well in the current releases (at least up to 5.1.42).

This query is quite fast: only 320 ms.

NOT EXISTS

Using this approach, the engine takes each product and checks if all of the relevant attributes satisfy the conditions. As soon as the first attribute failing the check is found, the condition is considered FALSE and further evaluation ceases.

This approach does not use aggregation (which is good), but it needs to browse all products and do random index seeks for each of the products (which is bad).

Here’s the query:

SELECT  COUNT(*)
FROM    t_product p
WHERE   NOT EXISTS
        (
        SELECT  NULL
        FROM    (
                SELECT  1 AS opt, 0 AS l, 100 AS h
                UNION ALL
                SELECT  2 AS opt, 10 AS l, 90 AS h
                UNION ALL
                SELECT  3 AS opt, 20 AS l, 80 AS h
                UNION ALL
                SELECT  4 AS opt, 30 AS l, 70 AS h
                UNION ALL
                SELECT  5 AS opt, 40 AS l, 60 AS h
                UNION ALL
                SELECT  6 AS opt, 50 AS l, 50 AS h
                ) v
        WHERE   NOT EXISTS
                (
                SELECT  NULL
                FROM    t_option o
                WHERE   o.product_id = p.id
                        AND o.option_id = opt
                        AND o.value BETWEEN l AND h
                )
        );

Field or reference '20100402_options.p.id' of SELECT #9 was resolved in SELECT #1
Field or reference 'v.opt' of SELECT #9 was resolved in SELECT #2
Field or reference 'v.l' of SELECT #9 was resolved in SELECT #2
Field or reference 'v.h' of SELECT #9 was resolved in SELECT #2
select count(0) AS `COUNT(*)` from `20100402_options`.`t_product` `p` where (not(exists(select NULL AS `NULL` from (select 1 AS `opt`,0 AS `l`,100 AS `h` union all select 2 AS `opt`,10 AS `l`,90 AS `h` union all select 3 AS `opt`,20 AS `l`,80 AS `h` union all select 4 AS `opt`,30 AS `l`,70 AS `h` union all select 5 AS `opt`,40 AS `l`,60 AS `h` union all select 6 AS `opt`,50 AS `l`,50 AS `h`) `v` where (not(exists(select NULL AS `NULL` from `20100402_options`.`t_option` `o` where ((`20100402_options`.`o`.`product_id` = `20100402_options`.`p`.`id`) and (`20100402_options`.`o`.`option_id` = `v`.`opt`) and (`20100402_options`.`o`.`value` between `v`.`l` and `v`.`h`))))))))

This query runs for 843 ms, or three times as long as its GROUP BY counterpart.

GROUP BY version seems to be the most efficient, but can we optimize it somehow?

Mixing two approaches

The main problem with the GROUP BY approach is, well, GROUP BY. In MySQL, it requires sorting or materialization (or both) which are quite expensive operations. Also, the recordset for each of the attributes is selected independently: no filter on one of the attributes affects the other ones. The filtering is only done after the aggregation.

The main problem with the NOT EXISTS approach is that every product should be checked for the conditions, though only a tiny fraction of all products satisfies at least several of them. Checking for conditions also requires index seeks in the blocks that are quite far away from each other which is not good for performance (especially if the index does not fit completely into the cache).

However, we can mix the two approaches.

Instead of using the product table as a source for all possible entities to check against the EAV table, we can take the condition on only one attribute that is the least likely to be satisfied.

Since entity/attribute pair form a natural primary key in the EAV table, filtering on an attribute is guaranteed to result in a set of unique entities. Each of these entities should then be further checked using the NOT EXISTS approach, but this time there are much fewer checks that need to be made, since with a properly chosen primary condition (which should be the most strong one) most of the non-matching entities are already filtered out.

This approach, on the one hand, does not require aggregation; on the other hand, the conditions are sieved rather then added up.

Let’s build the query:

SELECT  COUNT(*)
FROM    t_option oo
WHERE   oo.option_id = 6
        AND oo.value BETWEEN 50 AND 50
        AND NOT EXISTS
        (
        SELECT  NULL
        FROM    (
                SELECT  1 AS opt, 0 AS l, 100 AS h
                UNION ALL
                SELECT  2 AS opt, 10 AS l, 90 AS h
                UNION ALL
                SELECT  3 AS opt, 20 AS l, 80 AS h
                UNION ALL
                SELECT  4 AS opt, 30 AS l, 70 AS h
                UNION ALL
                SELECT  5 AS opt, 40 AS l, 60 AS h
                ) v
        WHERE   NOT EXISTS
                (
                SELECT  NULL
                FROM    t_option o
                WHERE   o.product_id = oo.product_id
                        AND o.option_id = opt
                        AND o.value BETWEEN l AND h
                )
        );

Field or reference '20100402_options.oo.product_id' of SELECT #8 was resolved in SELECT #1
Field or reference 'v.opt' of SELECT #8 was resolved in SELECT #2
Field or reference 'v.l' of SELECT #8 was resolved in SELECT #2
Field or reference 'v.h' of SELECT #8 was resolved in SELECT #2
select count(0) AS `COUNT(*)` from `20100402_options`.`t_option` `oo` where ((`20100402_options`.`oo`.`option_id` = 6) and (`20100402_options`.`oo`.`value` between 50 and 50) and (not(exists(select NULL AS `NULL` from (select 1 AS `opt`,0 AS `l`,100 AS `h` union all select 2 AS `opt`,10 AS `l`,90 AS `h` union all select 3 AS `opt`,20 AS `l`,80 AS `h` union all select 4 AS `opt`,30 AS `l`,70 AS `h` union all select 5 AS `opt`,40 AS `l`,60 AS `h`) `v` where (not(exists(select NULL AS `NULL` from `20100402_options`.`t_option` `o` where ((`20100402_options`.`o`.`product_id` = `20100402_options`.`oo`.`product_id`) and (`20100402_options`.`o`.`option_id` = `v`.`opt`) and (`20100402_options`.`o`.`value` between `v`.`l` and `v`.`h`)))))))))

As we can see, using the most selective condition as an initial row source for the further checks greatly improved the query speed.

The query time is now only 10 ms, or 30 times as fast as the GROUP BY approach.

Hope that helps.

I’m always glad to answer the questions regarding database queries.

Ask me a question

Suppose you’ve visited sites S0 … S50. All except S0 are 48% female; S0 is 100% male.

I’m guessing your gender, and I want to have a value close to 100%, not just the 49% that a straight average would give.

Also, consider that most demographics (i.e. everything other than gender) does not have the average at 50%. For example, the average probability of having kids 0-17 is ~37%.

The more a given site’s demographics are different from this average (e.g. maybe it’s a site for parents, or for child-free people), the more it should count in my guess of your status.

What’s the best way to calculate this?

This is a classical application of Bayes’ Theorem.

The formula to calculate the posterior probability is:

P(A|B) = P(B|A) × P(A) / P(B) = P(B|A) × P(A) / (P(B|A) × P(A) + P(B|A*) × P(A*))

, where:

• P(A|B) is the posterior probability of the visitor being a male (given that he visited the site)
• P(A) is the prior probability of the visitor being a male (initially, 50%)
• P(B) is the probability of (any Internet user) visiting the site
• P(B|A) is the probability of a user visiting the site, given that he is a male
• P(A*) is the prior probability of the visitor not being a male (initially, 50%)
• P(B|A*) is the probability of a user visiting the site, given that she is not a male.

Since a user can only be male or female:

P(A|B) = P(B|A)×P(A)/P(B) = P(B|A)×P(A) / (P(B|A)×P(A) + (1 - P(B|A))×(1 - P(A)))

P(B|A) is the number stored in the database (probability of the user being a male).

We consider the events of visiting the different sites to be independent (a fact that the user visited site A neither influences nor is influenced by the fact that the user also visited site B. This is of course not so, since the sites may exchange links etc., but we make this assumption for the sake of simplicity.

So, given a series of the sites, we take the initial probability (P0 = 0.5) and recursively substitute it into the following formula:

Pn = Sn×Pn-1 / (Sn×Pn-1 + (1 - Sn)×(1 - Pn-1))

Simple calculations show us the the recursion (which MySQL is not good at) can be replaced with an aggregate formula:

P = P0 * PROD(S) / (P0 * PROD(S) + (1 - P0) * PROD(1 - S))

, where PROD is the aggregate product of the sites’ probabilities of their visitors being a male.

SQL does not have a built-in aggregate product function, but it can be easily replaced by the aggregate sum on the logarithmic scale:

P = P0 * EXP(SUM(LN(S))) / (P0 * EXP(SUM(LN(S))) + (1 - P0) * EXP(SUM(LN(1 - S))))

Given all that, let’s create some sample tables:

Table creation details

CREATE TABLE filler (
        id INT NOT NULL PRIMARY KEY AUTO_INCREMENT
) ENGINE=Memory;

CREATE TABLE t_user (
        id INT NOT NULL PRIMARY KEY,
        name VARCHAR(20) NOT NULL,
        gender CHAR(1) NOT NULL
) ENGINE=InnoDB DEFAULT CHARSET=UTF8;

CREATE TABLE t_site (
        id INT NOT NULL PRIMARY KEY,
        name VARCHAR(20) NOT NULL,
        male DOUBLE NOT NULL
) ENGINE=InnoDB DEFAULT CHARSET=UTF8;

CREATE TABLE t_visit (
        u_id INT NOT NULL,
        s_id INT NOT NULL,
        PRIMARY KEY (u_id, s_id)
) ENGINE=InnoDB DEFAULT CHARSET=UTF8;

DELIMITER $$

CREATE PROCEDURE prc_filler(cnt INT)
BEGIN
        DECLARE _cnt INT;
        SET _cnt = 1;
        WHILE _cnt <= cnt DO
                INSERT
                INTO    filler
                SELECT  _cnt;
                SET _cnt = _cnt + 1;
        END WHILE;
END
$$

DELIMITER ;

START TRANSACTION;
CALL prc_filler(1000);
COMMIT;

INSERT
INTO    t_user
SELECT  id, CONCAT('User ', id),
        CASE WHEN RAND(20100325) > 0.5 THEN 'M' ELSE 'F' END
FROM    filler;

INSERT
INTO    t_site
SELECT  id, CONCAT('Site ', id),
        RAND(20100325 << 1) * 0.94 + 0.03
FROM    filler;

INSERT
INTO    t_visit
SELECT  u_id, s_id
FROM    (
        SELECT  u.id AS u_id, s.id AS s_id, u.gender, s.male,
                RAND(20100325 << 2) AS rnds,
                RAND(20100325 << 3) AS rndm
        FROM    t_user u
        CROSS JOIN
                t_site s
        ) q
WHERE   rnds < 0.05
        AND rndm < CASE gender WHEN 'M' THEN male ELSE 1 - male END;

There are 1,000 users and 1,000 sites. The sites are assigned with maleness from 0.03 to 0.97.

User randomly visit the sites according to their gender and the site gender distribution. There are 25 visits per user in average.

Let’s try to guess the users’ gender and return only wrong guesses.

We will assume that the user is male when the posterior probability of the user being male is more than 0.99, female if that is less than 0.01, undefined if within 0.01 and 0.99:

SELECT  *, CASE WHEN posterior < 0.01 THEN 'F' WHEN posterior > 0.99 THEN 'M' ELSE 'U' END AS guessed
FROM    (
        SELECT  u.*,
                prior * EXP(SUM(LN(male))) / (prior * EXP(SUM(LN(male))) + (1 - prior) * EXP(SUM(LN(1 - male)))) AS posterior
        FROM    (
                SELECT  0.5 AS prior
                ) vars
        CROSS JOIN
                t_user u
        LEFT JOIN
                t_visit v
        ON      v.u_id = u.id
        LEFT JOIN
                t_site s
        ON      s.id = v.s_id
        GROUP BY
                u.id
        ) q
HAVING  guessed <> gender

select `q`.`id` AS `id`,`q`.`name` AS `name`,`q`.`gender` AS `gender`,`q`.`posterior` AS `posterior`,(case when (`q`.`posterior` < 0.01) then 'F' when (`q`.`posterior` > 0.99) then 'M' else 'U' end) AS `guessed` from (select `20100325_bayes`.`u`.`id` AS `id`,`20100325_bayes`.`u`.`name` AS `name`,`20100325_bayes`.`u`.`gender` AS `gender`,(('0.5' * exp(sum(ln(`20100325_bayes`.`s`.`male`)))) / (('0.5' * exp(sum(ln(`20100325_bayes`.`s`.`male`)))) + ((1 - '0.5') * exp(sum(ln((1 - `20100325_bayes`.`s`.`male`))))))) AS `posterior` from (select 0.5 AS `prior`) `vars` join `20100325_bayes`.`t_user` `u` left join `20100325_bayes`.`t_visit` `v` on((`20100325_bayes`.`v`.`u_id` = `20100325_bayes`.`u`.`id`)) left join `20100325_bayes`.`t_site` `s` on((`20100325_bayes`.`s`.`id` = `20100325_bayes`.`v`.`s_id`)) where 1 group by `20100325_bayes`.`u`.`id`) `q` having (convert(`guessed` using utf8) <> `q`.`gender`)

From 1,000 users, we only have 14 results outside the credible interval of 99% and all of these are undefined rather than false.

Vlad Enache asks:

In MySQL I have a table called meanings with three columns:

meanings

person	word	meaning
1	1	4
1	2	19
1	2	7
1	3	5

word has 16 possible values, meaning has 26.

A person assigns one or more meanings to each word. In the sample above, person 1 assigned two meanings to word 2.

There will be thousands of persons.

I need to find the top three meanings for each of the 16 words, with their frequencies. Something like:

• word 1: meaning 5 (35% of people), meaning 19 (22% of people), meaning 2 (13% of people)
• word 2: meaning 8 (57%), meaning 1 (18%), meaning 22 (7%)

etc.

Is it possible to solve this with a single MySQL query?

This task is a typical greatest-n-per-group problem.

Earlier I described some solutions to it, one using session variables, and another one using LIMIT in a subquery. However, these solutions imply that the records are taken from a single table, while this task needs to retrieve three greatest aggregates, not three greatest records. It is not recommended to mix the session variables with the JOINs, and using the LIMIT solution would be inefficient, since the aggregates can not be indexed.

Some databases, namely, PostgreSQL, used to exploit the array functionality for this task (before the window functions were introduced in 8.4).

Unfortunately, MySQL does not support arrays, but we can emulate this behavior using string functions and GROUP_CONCAT.

Let’s create a sample table:

Table creation details

CREATE TABLE filler (
        id INT NOT NULL PRIMARY KEY AUTO_INCREMENT
) ENGINE=Memory;

CREATE TABLE t_meaning (
        person INT NOT NULL,
        word INT NOT NULL,
        meaning INT NOT NULL,
        PRIMARY KEY (person, word, meaning),
        KEY ix_meaning_word_meaning (word, meaning)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

DELIMITER $$

CREATE PROCEDURE prc_filler(cnt INT)
BEGIN
        DECLARE _cnt INT;
        SET _cnt = 1;
        WHILE _cnt <= cnt DO
                INSERT
                INTO    filler
                SELECT  _cnt;
                SET _cnt = _cnt + 1;
        END WHILE;
END
$$

DELIMITER ;

START TRANSACTION;
CALL prc_filler(1000000);
COMMIT;

INSERT
INTO    t_meaning (person, word, meaning)
SELECT  (id - 1) div 32 + 1,
        (id - 1) % 16 + 1,
        ((((id - 1) div 16) % 2) * 13) + CEILING(RAND(20100318 << 1) * 13)
FROM    filler;

The table contains 1,000,000 records for 32,500 people, each giving two random meanings to each of the 16 words.

All three fields comprise a PRIMARY KEY, and there is an additional index on (word, meaning).

Here’s what we need to do to build the query:

1. Calculate the count of records with each meaning for each word, using GROUP BY.

   This will use the secondary index on (word, meaning) that we created above to avoid sorting (using filesort) or materialization (using temporary)

2. GROUP_CONCAT the concatenated meaning and count from the previous recordset, ordering the values by count in descending order.

   Note that theoretically this could result in very large strings which would not fit into @@group_concat_max_len and get truncated. But we can use it nevertheless: first, there are only 26 meanings per word possible, second, of these 26 meanings we only the first three.

3. Join the resultset with the dummy table containing 3 records. To do this, we will just concatenate 3 one-record SELECT statement using UNION ALL.

   This is not a very elegant solution, since it does not allow parameterizing the n for the query, but MySQL does not support dummy recordset generation functions like generate_series.

   Alternatively, one could build a dummy table and use a LIMIT to select top values from it. But for the sake of simplicity, I will go with the UNION ALL solution.

4. Extract the individual values from the concatenated string using SUBSTRING_INDEX

5. Calculate the ratios, using the number of distinct persons as a total.

   This way, ratio of each word’s meaning will be computed correctly, though the sum of ratios can exceed 100% (since one person can give more than one meaning to a word).

And here’s the query:

SELECT  word, 4 - lvl AS position, SUBSTRING_INDEX(rec, ':', 1) AS meaning, SUBSTRING_INDEX(rec, ':', -1) / total AS rate
FROM    (
        SELECT  word, total, SUBSTRING_INDEX(SUBSTRING_INDEX(tops, ',', -lvl), ',', 1) AS rec, lvl
        FROM    (
                SELECT  word, total, SUBSTRING_INDEX(ms, ',', 3) AS tops, lvl
                FROM    (
                        SELECT  word,
                                (
                                SELECT  COUNT(*)
                                FROM    (
                                        SELECT  DISTINCT person
                                        FROM    t_meaning
                                        ) dp
                                ) AS total,
                                CAST(GROUP_CONCAT(CONCAT_WS(':', meaning, cnt) ORDER BY cnt DESC, meaning DESC) AS CHAR) AS ms
                        FROM    (
                                SELECT  word, meaning, COUNT(*) AS cnt
                                FROM    t_meaning
                                GROUP BY
                                        word, meaning
                                ) q
                        GROUP BY
                                word
                        ) q2
                CROSS JOIN
                        (
                        SELECT  1 AS lvl
                        UNION ALL
                        SELECT  2 AS lvl
                        UNION ALL
                        SELECT  3 AS lvl
                        ) fields
                ) q3
        ) q4
ORDER BY
        word, position

select `q4`.`word` AS `word`,(4 - `q4`.`lvl`) AS `position`,substring_index(`q4`.`rec`,':',1) AS `meaning`,(substring_index(`q4`.`rec`,':',-(1)) / `q4`.`total`) AS `rate` from (select `q3`.`word` AS `word`,`q3`.`total` AS `total`,substring_index(substring_index(`q3`.`tops`,',',-(`q3`.`lvl`)),',',1) AS `rec`,`q3`.`lvl` AS `lvl` from (select `q2`.`word` AS `word`,`q2`.`total` AS `total`,substring_index(`q2`.`ms`,',',3) AS `tops`,`fields`.`lvl` AS `lvl` from (select `q`.`word` AS `word`,(select count(0) AS `COUNT(*)` from (select distinct `20100318_word`.`t_meaning`.`person` AS `person` from `20100318_word`.`t_meaning`) `dp`) AS `total`,cast(group_concat(concat_ws(':',`q`.`meaning`,`q`.`cnt`) order by `q`.`cnt` DESC,`q`.`meaning` DESC separator ',') as char charset latin1) AS `ms` from (select `20100318_word`.`t_meaning`.`word` AS `word`,`20100318_word`.`t_meaning`.`meaning` AS `meaning`,count(0) AS `cnt` from `20100318_word`.`t_meaning` group by `20100318_word`.`t_meaning`.`word`,`20100318_word`.`t_meaning`.`meaning`) `q` group by `q`.`word`) `q2` join (select 1 AS `lvl` union all select 2 AS `lvl` union all select 3 AS `lvl`) `fields`) `q3`) `q4` order by `q4`.`word`,(4 - `q4`.`lvl`)

Note that we used

SELECT  COUNT(*)
FROM    (
        SELECT  DISTINCT person
        FROM    t_meaning
        ) q

instead of more simple

SELECT  COUNT(DISTINCT person)
FROM    t_meaning

The latter looks more elegant, but MySQL is not able to use INDEX FOR GROUP BY for it.

That’s why we used the former subquery to calculate the totals, and we can see in the query plan that the INDEX FOR GROUP BY is being used.

The overall query time is 1.10 sec which is just a little more than the time required to iterate all records in the table.

Hope that helps.

I’m always glad to answer the questions regarding database queries.

Ask me a question

I have a table product with products and table sale with all sale operations that were done on these products.

I would like to get 10 most often sold products today and what I did is this:

SELECT  p.*, COUNT(s.id) AS sumsell
FROM    product p
LEFT JOIN
        sale s
ON      s.product_id = p.id
        AND s.dt >= '2010-01-01'
        AND s.dt < '2010-01-02'
GROUP BY
        p.id
ORDER BY
        sumsell DESC
LIMIT 10

, but performance of it is very slow.

What can I do to increase performance of this particular query?

The query involves a LEFT JOIN which in MySQL world means that products will be made leading in the query. Each record of product will be taken and checked against sale table to find out the number of matching records. If no matching records are found, 0 is returned.

Let’s create the sample tables:

Table creation details

CREATE TABLE filler (
        id INT NOT NULL PRIMARY KEY AUTO_INCREMENT
) ENGINE=Memory;

DELIMITER $$

CREATE TABLE product (
        id INT NOT NULL PRIMARY KEY,
        name VARCHAR(30) NOT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

CREATE TABLE sale (
        id INT NOT NULL PRIMARY KEY,
        product_id INT NOT NULL,
        amount FLOAT NOT NULL,
        dt DATETIME NOT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

CREATE PROCEDURE prc_filler(cnt INT)
BEGIN
        DECLARE _cnt INT;
        SET _cnt = 1;
        WHILE _cnt <= cnt DO
                INSERT
                INTO    filler
                SELECT  _cnt;
                SET _cnt = _cnt + 1;
        END WHILE;
END
$$

DELIMITER ;

START TRANSACTION;
CALL prc_filler(500000);
COMMIT;

INSERT
INTO    product
SELECT  id, CONCAT('Product ', id)
FROM    filler;

INSERT
INTO    sale (id, product_id, amount, dt)
SELECT  id,
        FLOOR(RAND(20100305) * 500000) + 1,
        RAND(20100305 << 1) * 100 + 1,
        '2010-03-06' - INTERVAL id MINUTE
FROM    filler;

CREATE INDEX ix_sale_product_dt ON sale (product_id, dt);
CREATE INDEX ix_sale_dt_product ON sale (dt, product_id);

The table contains 500,000 products and 500,000 random sales (1,440 sales per day).

Now, let’s run the query similar to the author’s one. I adjusted the period so that fewer than 10 actual sales were made during the period and LEFT JOIN records can be seen in the table:

SELECT  p.*, COUNT(s.id) AS sumsell
FROM    product p
LEFT JOIN
        sale s
ON      s.product_id = p.id
        AND s.dt >= '2010-01-01'
        AND s.dt < '2010-01-01 00:07:00'
GROUP BY
        p.id
ORDER BY
        sumsell DESC, p.id
LIMIT 10

View query details

select `20100305_left`.`p`.`id` AS `id`,`20100305_left`.`p`.`name` AS `name`,count(`20100305_left`.`s`.`id`) AS `sumsell` from `20100305_left`.`product` `p` left join `20100305_left`.`sale` `s` on(((`20100305_left`.`s`.`product_id` = `20100305_left`.`p`.`id`) and (`20100305_left`.`s`.`dt` >= '2010-01-01') and (`20100305_left`.`s`.`dt` < '2010-01-01 00:07:00'))) where 1 group by `20100305_left`.`p`.`id` order by count(`20100305_left`.`s`.`id`) desc,`20100305_left`.`p`.`id` limit 10

The query runs for 3 seconds.

We see that, first, product is made leading, and, second, only a part of the index on sale (product, dt) is used: each sale is only filtered on product, not on date.

Since there were only 7 sales during the period we have chosen, it would be a wise decision to make sale leading in the join so that it could be filtered on date and the resulting recordset was then joined to product. This would result in at most 7 PRIMARY KEY seeks instead of 500,000 range scans and would be much more efficient.

However, this is only possible with the INNER JOIN, and if there are less then 10 products sold within the time period, we will not see the rest.

To work around this, we need to emulate the LEFT JOIN:

1. Find the products sold within the time period, using an INNER JOIN of product with the resultset containg aggregated sales.

2. Find the products not sold within the time period, using NOT EXISTS predicate.

3. Concatenate the two resultsets using UNION ALL.

The step 2 implies that product is leading again, so normally it would not be much of improvement. But in our case, we don’t need the whole recordset, we only need the top 10 sales.

So we can just order and limit the recordsets retrieved on steps 1 and 2 to ten records each, concatenate them, then order and limit the resulting recordset again to ten records.

The second resultset will contain a hardcoded 0 in the sumsell, so we just need to order it on product.id. Since product is an InnoDB table and product.id is a clustered PRIMARY KEY, this is not a problem.

Here’s the query:

SELECT  p.*, sumsell
FROM    (
        SELECT  *
        FROM    (
                SELECT  product_id, sumsell
                FROM    (
                        SELECT  product_id, COUNT(*) AS sumsell
                        FROM    sale si
                        WHERE   dt >= '2010-01-01'
                                AND dt < '2010-01-01 00:07:00'
                        GROUP BY
                                product_id
                        ) si
                ORDER BY
                        sumsell DESC, product_id
                LIMIT 10
                ) q1
        UNION ALL
        SELECT  *
        FROM    (
                SELECT  p.id, 0
                FROM    product p
                WHERE   NOT EXISTS
                        (
                        SELECT  NULL
                        FROM    sale si
                        WHERE   product_id = p.id
                                AND dt >= '2010-01-01'
                                AND dt < '2010-01-01 00:07:00'
                        )
                ORDER BY
                        p.id
                LIMIT 10
                ) q2
        ORDER BY
                sumsell DESC, product_id
        LIMIT 10
        ) q
JOIN    product p
ON      p.id = q.product_id

View query details

Field or reference '20100305_left.p.id' of SELECT #7 was resolved in SELECT #6
select `20100305_left`.`p`.`id` AS `id`,`20100305_left`.`p`.`name` AS `name`,`q`.`sumsell` AS `sumsell` from (select `q1`.`product_id` AS `product_id`,`q1`.`sumsell` AS `sumsell` from (select `si`.`product_id` AS `product_id`,`si`.`sumsell` AS `sumsell` from (select `20100305_left`.`si`.`product_id` AS `product_id`,count(0) AS `sumsell` from `20100305_left`.`sale` `si` where ((`20100305_left`.`si`.`dt` >= '2010-01-01') and (`20100305_left`.`si`.`dt` < '2010-01-01 00:07:00')) group by `20100305_left`.`si`.`product_id`) `si` order by `si`.`sumsell` desc,`si`.`product_id` limit 10) `q1` union all select `q2`.`id` AS `id`,`q2`.`0` AS `0` from (select `20100305_left`.`p`.`id` AS `id`,0 AS `0` from `20100305_left`.`product` `p` where (not(exists(select NULL AS `NULL` from `20100305_left`.`sale` `si` where ((`20100305_left`.`si`.`product_id` = `20100305_left`.`p`.`id`) and (`20100305_left`.`si`.`dt` >= '2010-01-01') and (`20100305_left`.`si`.`dt` < '2010-01-01 00:07:00'))))) order by `20100305_left`.`p`.`id` limit 10) `q2` order by `sumsell` desc,`product_id` limit 10) `q` join `20100305_left`.`product` `p` where (`20100305_left`.`p`.`id` = `q`.`product_id`)

This query completes in less than 7 ms (which is comparable to the time measurement error).