#1. A little theory

Core of a snowflake is six large symmetrical ice crystals growing from the common center. Out of these larger crystals other, smaller, crystals grow.

The overall shape of the snowflake is defined by how large do crystals grow and where exactly are they attached to each other.

These things are defined by fluctuations in air and temperature conditions around the snowflake. Because the flake itself is very small, in any given moment the conditions are nearly identical around each crystal, that’s why the offspring crystals pop up in almost same places and grow to almost same lengths. Different flakes, though, constantly move to and from each other and are subject to very different fluctuations, and that’s why they grow so unique.

Except for the root crystals (of which there are six), the child icicles grow in symmetrical pairs. More than that, each branch grows their own children (also in pairs), so on each step there are twice as many crystals, but they all share almost same length and angle. This gives the snowflake its symmetrical look.

So we can easily see that, despite the fact there may be many child crystals, the shape of a snowflake is defined by a relatively small number of parameters: how many children each crystal produces, where are they attached to it, at which angle they grow and to which length.

Now, let’s try to model it.

#2. Defining parameters

To begin with, we will assume the length of the initial larger crystal as 1, and its angle as 0. Later we may easily scale and rotate the crystal.

There can be any number of child crystals (or, rather, pairs of child crystals), each pair attaching to its parent in a random place and growing at random angle. However, this places and angles, though random, are shared across the snowflake: if one pair grows here and at this angle, the similar pairs will grow on the same place and at the same angle on their parents.

This means that there are 6 * (2 ^ level) twins of each crystal, where level is the number of its ancestors. There are 6 root crystals, 12 first-level crystals in the first first-level pairs (each growing in the same place at the same angle to the same length on root crystals), 12 first-level crystals in the second first-level pairs (again, sharing all parameters with themselves but not with the first pairs), 24 first-first grandchildren, 24 first-second grandchildren etc.

So, to define the snowflake shape, we need to define the number of pairs on each step, and for each pair define its place on the parent crystal, angle and length.

These parameters are random but have some constraints. Child crystals are shorter than their parents; they grow at sharp angles, and the number of pairs is limited. Since the parameters of each crystal depend on its parent, we would need a recursive query for that:

SELECT  SETSEED(0.20111231);

WITH    RECURSIVE
        params (id, cut, len, alpha, level, spikes) AS
        (
        SELECT  ARRAY[0::INT], 0::DOUBLE PRECISION, 1::DOUBLE PRECISION, 0::DOUBLE PRECISION, 1, FLOOR(RANDOM() * 4)::INT AS spikes
        UNION ALL
        SELECT  id, cut, len, alpha, level, FLOOR(RANDOM() * 4)::INT AS spikes
        FROM    (
                SELECT  id || generate_series(0, spikes) AS id,
                        RANDOM() AS cut, len * RANDOM() * 0.5 AS len, (PI() / 2) * (0.3 + RANDOM() * 0.3) AS alpha, level + 1 AS level
                FROM    params
                ) q
        WHERE   level <= 3
        )
SELECT  id::TEXT, cut, len, alpha, level, spikes
FROM    params;

Here, id is an array defining the path to the crystal pairs: {0} is the root branch, {0, 0} is the first child pair, {0, 1, 0} is the first child of the second child etc.

cut defines the position where the crystals are attached to the parent: 0.356672627385706 in the second row means the first child will grow at 35.6% of the root branches’ length.

len is the child length (in respect to the root branch, not immediate parent), angle is the angle at which they grow (in respect to the immediate parent), and spikes is the number of children pairs for the given crystal. level, I hope, is self-explanatory.

#3. Defining crystal coordinates

Now that we have the shape of our flake defined we need to build the actual coordinates for each crystal. To do this we again would need a recursive query.

First, we should make a set of branches on each level. We would start from a single root branch (it will be easy to clone it later) and generate a set of branches on each level. Each record in this set would correspond to an actual snowflake crystal:

WITH    RECURSIVE
        params (id, cut, len, alpha, level, spikes) AS
        (
        SELECT  ARRAY[0::INT], 0::DOUBLE PRECISION, 1::DOUBLE PRECISION, 0::DOUBLE PRECISION, 1, FLOOR(RANDOM() * 4)::INT AS spikes
        UNION ALL
        SELECT  id, cut, len, alpha, level, FLOOR(RANDOM() * 4)::INT AS spikes
        FROM    (
                SELECT  id || generate_series(0, spikes) AS id,
                        RANDOM() AS cut, len * RANDOM() * 0.5 AS len, (PI() / 2) * (0.3 + RANDOM() * 0.3) AS alpha, level + 1 AS level
                FROM    params
                ) q
        WHERE   level <= 3
        ),
        tree AS
        (
        SELECT  *, generate_series(0, (1 << (level - 1)) - 1) AS branch
        FROM    params
        )
SELECT  id::TEXT, cut, len, alpha, level, spikes, branch
FROM    tree;

View query output

This is just a copy of the parameters recordset with each parameter duplicated as many times as the crystal level requires. There is one root branch, two copies of level 1 branches, four copies of level 2 branches etc. Each instance is defined by the parameter branch.

To calculate coordinates of each crystal we need to traverse to it from the top.

The coordinates of the root branch are known: they are (0, 0), (1, 0) (by definition).

To build the first pair (two crystals, each 28.5% long, growing at 35.6% from the beginning of the root branch at angles 51° 18′ and −51° 18′, respectively) we would need to take the coordinates of the parent, find the start point (it would be (Xs + (Xe - Xs) * cut, Ys + (Ye - Ys) * cut) and then the coordinates of the end point (by adding len * COS(α) and len * SIN(α) to X and Y of the start point, respectively). α in this formula is in respect to the coordinate grid, not to the parent, and to find it we just need to sum all angles from the ancestors.

This is best done with another recursive query. On each step we should find immediate children of the current parent.

This can be easily achieved using the following join condition: child.id > parent.id AND child.id <= parend.id || spikes AND ARRAY_LENGTH(child.level, 1) = ARRAY_LENGTH(parent.level, 1) + 1. This condition employs PostgreSQL array arithmetics: if we have parent.id = {0, 2, 1} with 3 children (spikes = 2), then the condition would return {0, 2, 1, 0}, {0, 2, 1, 1} and {0, 2, 1, 2}. This hierarchy model is called materialized path, and some day I will write a post about it.

Since the id alone defines the parameters, not the instance, we need to add once more condition to find the instances. Crystals 0 and 1 would have child pairs 0, 1 and 2, 3, respectively, so we'll include child.branch BETWEEN p.branch * 2 AND child.branch * 2 + 1 into the join condition.

One more thing to do is to find whether we should add positive or negative angle. It's simple: even branches are negative, odd ones are positive.

And here's the query:

SELECT  SETSEED(0.20111231);

WITH    RECURSIVE
        params (id, cut, len, alpha, level, spikes) AS
        (
        SELECT  ARRAY[0::INT], 0::DOUBLE PRECISION, 1::DOUBLE PRECISION, 0::DOUBLE PRECISION, 1, FLOOR(RANDOM() * 4)::INT AS spikes
        UNION ALL
        SELECT  id, cut, len, alpha, level, FLOOR(RANDOM() * 4)::INT AS spikes
        FROM    (
                SELECT  id || generate_series(0, spikes) AS id,
                        RANDOM() AS cut, len * RANDOM() * 0.5 AS len, (PI() / 2) * (0.3 + RANDOM() * 0.3) AS alpha, level + 1 AS level
                FROM    params
                ) q
        WHERE   level <= 3
        ),
        tree AS
        (
        SELECT  *, generate_series(0, (1 << (level - 1)) - 1) AS branch
        FROM    params
        ),
        points AS
        (
        SELECT  id,
                0::double precision AS xs,
                0::double precision AS ys,
                len AS xe,
                0::double precision AS ye,
                0::double precision AS alpha,
                branch,
                level,
                spikes
        FROM    tree
        WHERE   id = ARRAY[0]
        UNION ALL
        SELECT  id,
                xs + (xe - xs) * cut AS xs,
                ys + (ye - ys) * cut AS ys,
                xs + (xe - xs) * cut + len * COS(alpha) AS xe,
                ys + (ye - ys) * cut + len * SIN(alpha) AS ye,
                alpha,
                branch,
                level,
                spikes
        FROM    (
                SELECT  t.id, p.id || t.id AS uid, xs, xe, ys, ye, cut, len,
                        p.alpha + t.alpha * ((t.branch % 2) * 2 - 1) AS alpha,
                        t.branch, t.level, t.spikes
                FROM    points p
                JOIN    tree t
                ON      t.id > p.id
                        AND t.id <= (p.id || p.spikes)
                        AND ARRAY_LENGTH(t.id, 1) = p.level + 1
                        AND t.branch BETWEEN p.branch * 2 AND p.branch * 2 + 1
                ) q
        )
SELECT  id::TEXT, xs, ys, xe, ye, alpha, branch, level
FROM    points

View query output

Now, we have coordinates for each crystal.

#4. Visualizing

The most tedious part about SQL graphics is visualizing them. To do this, we'll employ PostgreSQL's geometrical functions.

Each crystal can be represented as a path between its start and end points. This can be used by constructing a line segment (LSEG(POINT, POINT)) using two point constructors (POINT(DOUBLE PRECISION, DOUBLE PRECISION)) and converting it to a path. Unfortunately, PostgreSQL does not allow direct lseg to path conversion but the latter can be easily constructed from the TEXT representation of an lseg.

We have six root branches so each crystal should be cloned to make six copies. It's easiest to make it by using PATH * POINT operator: it rotates and scales the PATH around (0, 0) so that (1, 0) becomes POINT. To construct the points, we will generate six rotation angles with step of 60° and will multiply the path by POINT(COS(α), SIN(α)). These multiplications will preserve lengths.

Finally we need to actually display the snowflake on the screen. To do this, we will generate a set of 80 × 80 records(x and y), defining the grid from (-1, -1) to (1, 1) with step of 1/40 units. Then we'll see if there is at least one crystal within distance of 1/40 units from each cell on the grid (using POINT <-> PATH distance operator and EXISTS). If there is, we will return a number sign (#) for this cell, otherwise a space.

Then we'll group the cells by lines (y), concatenate columns (using ARRAY_AGG and ARRAY_TO_STRING) and output the lines.

#5. The snowflake

And here's our snowflake:

SELECT  SETSEED(0.20111231);

WITH    RECURSIVE
        params (id, cut, len, alpha, level, spikes) AS
        (
        SELECT  ARRAY[0::INT], 0::DOUBLE PRECISION, 1::DOUBLE PRECISION, 0::DOUBLE PRECISION, 1, FLOOR(RANDOM() * 4)::INT AS spikes
        UNION ALL
        SELECT  id, cut, len, alpha, level, FLOOR(RANDOM() * 4)::INT AS spikes
        FROM    (
                SELECT  id || generate_series(0, spikes) AS id,
                        RANDOM() AS cut, len * RANDOM() * 0.5 AS len, (PI() / 2) * (0.3 + RANDOM() * 0.3) AS alpha, level + 1 AS level
                FROM    params
                ) q
        WHERE   level <= 3
        ),
        tree AS
        (
        SELECT  *, generate_series(0, (1 << (level - 1)) - 1) AS branch
        FROM    params
        ),
        points AS
        (
        SELECT  id,
                0::double precision AS xs,
                0::double precision AS ys,
                len AS xe,
                0::double precision AS ye,
                0::double precision AS alpha,
                branch,
                level,
                spikes
        FROM    tree
        WHERE   id = ARRAY[0]
        UNION ALL
        SELECT  id,
                xs + (xe - xs) * cut AS xs,
                ys + (ye - ys) * cut AS ys,
                xs + (xe - xs) * cut + len * COS(alpha) AS xe,
                ys + (ye - ys) * cut + len * SIN(alpha) AS ye,
                alpha,
                branch,
                level,
                spikes
        FROM    (
                SELECT  t.id, p.id || t.id AS uid, xs, xe, ys, ye, cut, len,
                        p.alpha + t.alpha * ((t.branch % 2) * 2 - 1) AS alpha,
                        t.branch, t.level, t.spikes
                FROM    points p
                JOIN    tree t
                ON      t.id > p.id
                        AND t.id <= (p.id || p.spikes)
                        AND ARRAY_LENGTH(t.id, 1) = p.level + 1
                        AND t.branch BETWEEN p.branch * 2 AND p.branch * 2 + 1
                ) q
        ),
        lines AS
        (
        SELECT  PATH(LSEG(POINT(xs, ys), POINT(xe, ye))::TEXT) * POINT(COS(RADIANS(turn)), SIN(RADIANS(turn))) AS line,
                *
        FROM    points
        CROSS JOIN
                generate_series(0, 300, 60) AS turn
        )
SELECT  ARRAY_TO_STRING
                (
                ARRAY_AGG
                        (
                        CASE
                                EXISTS
                                (
                                SELECT  NULL
                                FROM    lines
                                WHERE   POINT(x::DOUBLE PRECISION / scale, y::DOUBLE PRECISION / scale) <-> line <= (0.7 / scale)
                                )
                        WHEN    TRUE THEN
                                '#'
                        ELSE    ' '
                        END
                        ORDER BY
                                x
                        ),
                ''
                )
FROM    (
        SELECT  *,
                generate_series(-scale, scale - 1) AS y
        FROM    (
                SELECT  scale, generate_series(-scale, scale - 1) AS x
                FROM    (
                        VALUES
                        (40)
                        ) q (scale)
                ) q
        ) q
GROUP BY
        y
ORDER BY
        y

#6. Some more unique snowflakes

View query

SELECT  SETSEED(0.201112311);

WITH    RECURSIVE
        params (id, cut, len, alpha, level, spikes) AS
        (
        SELECT  ARRAY[0::INT], 0::DOUBLE PRECISION, 1::DOUBLE PRECISION, 0::DOUBLE PRECISION, 1, FLOOR(RANDOM() * 4)::INT AS spikes
        UNION ALL
        SELECT  id, cut, len, alpha, level, FLOOR(RANDOM() * 4)::INT AS spikes
        FROM    (
                SELECT  id || generate_series(0, spikes) AS id,
                        RANDOM() AS cut, len * RANDOM() * 0.5 AS len, (PI() / 2) * (0.3 + RANDOM() * 0.3) AS alpha, level + 1 AS level
                FROM    params
                ) q
        WHERE   level <= 3
        ),
        tree AS
        (
        SELECT  *, generate_series(0, (1 << (level - 1)) - 1) AS branch
        FROM    params
        ),
        points AS
        (
        SELECT  id,
                0::double precision AS xs,
                0::double precision AS ys,
                len AS xe,
                0::double precision AS ye,
                0::double precision AS alpha,
                branch,
                level,
                spikes
        FROM    tree
        WHERE   id = ARRAY[0]
        UNION ALL
        SELECT  id,
                xs + (xe - xs) * cut AS xs,
                ys + (ye - ys) * cut AS ys,
                xs + (xe - xs) * cut + len * COS(alpha) AS xe,
                ys + (ye - ys) * cut + len * SIN(alpha) AS ye,
                alpha,
                branch,
                level,
                spikes
        FROM    (
                SELECT  t.id, p.id || t.id AS uid, xs, xe, ys, ye, cut, len,
                        p.alpha + t.alpha * ((t.branch % 2) * 2 - 1) AS alpha,
                        t.branch, t.level, t.spikes
                FROM    points p
                JOIN    tree t
                ON      t.id > p.id
                        AND t.id <= (p.id || p.spikes)
                        AND ARRAY_LENGTH(t.id, 1) = p.level + 1
                        AND t.branch BETWEEN p.branch * 2 AND p.branch * 2 + 1
                ) q
        ),
        lines AS
        (
        SELECT  PATH(LSEG(POINT(xs, ys), POINT(xe, ye))::TEXT) * POINT(COS(RADIANS(turn)), SIN(RADIANS(turn))) AS line,
                *
        FROM    points
        CROSS JOIN
                generate_series(0, 300, 60) AS turn
        )
SELECT  ARRAY_TO_STRING
                (
                ARRAY_AGG
                        (
                        CASE
                                EXISTS
                                (
                                SELECT  NULL
                                FROM    lines
                                WHERE   POINT(x::DOUBLE PRECISION / scale, y::DOUBLE PRECISION / scale) <-> line <= (0.7 / scale)
                                )
                        WHEN    TRUE THEN
                                '#'
                        ELSE    ' '
                        END
                        ORDER BY
                                x
                        ),
                ''
                )
FROM    (
        SELECT  *,
                generate_series(-scale, scale - 1) AS y
        FROM    (
                SELECT  scale, generate_series(-scale, scale - 1) AS x
                FROM    (
                        VALUES
                        (40)
                        ) q (scale)
                ) q
        ) q
GROUP BY
        y
ORDER BY
        y

View query

SELECT  SETSEED(0.201112312);

WITH    RECURSIVE
        params (id, cut, len, alpha, level, spikes) AS
        (
        SELECT  ARRAY[0::INT], 0::DOUBLE PRECISION, 1::DOUBLE PRECISION, 0::DOUBLE PRECISION, 1, FLOOR(RANDOM() * 4)::INT AS spikes
        UNION ALL
        SELECT  id, cut, len, alpha, level, FLOOR(RANDOM() * 4)::INT AS spikes
        FROM    (
                SELECT  id || generate_series(0, spikes) AS id,
                        RANDOM() AS cut, len * RANDOM() * 0.5 AS len, (PI() / 2) * (0.3 + RANDOM() * 0.3) AS alpha, level + 1 AS level
                FROM    params
                ) q
        WHERE   level <= 3
        ),
        tree AS
        (
        SELECT  *, generate_series(0, (1 << (level - 1)) - 1) AS branch
        FROM    params
        ),
        points AS
        (
        SELECT  id,
                0::double precision AS xs,
                0::double precision AS ys,
                len AS xe,
                0::double precision AS ye,
                0::double precision AS alpha,
                branch,
                level,
                spikes
        FROM    tree
        WHERE   id = ARRAY[0]
        UNION ALL
        SELECT  id,
                xs + (xe - xs) * cut AS xs,
                ys + (ye - ys) * cut AS ys,
                xs + (xe - xs) * cut + len * COS(alpha) AS xe,
                ys + (ye - ys) * cut + len * SIN(alpha) AS ye,
                alpha,
                branch,
                level,
                spikes
        FROM    (
                SELECT  t.id, p.id || t.id AS uid, xs, xe, ys, ye, cut, len,
                        p.alpha + t.alpha * ((t.branch % 2) * 2 - 1) AS alpha,
                        t.branch, t.level, t.spikes
                FROM    points p
                JOIN    tree t
                ON      t.id > p.id
                        AND t.id <= (p.id || p.spikes)
                        AND ARRAY_LENGTH(t.id, 1) = p.level + 1
                        AND t.branch BETWEEN p.branch * 2 AND p.branch * 2 + 1
                ) q
        ),
        lines AS
        (
        SELECT  PATH(LSEG(POINT(xs, ys), POINT(xe, ye))::TEXT) * POINT(COS(RADIANS(turn)), SIN(RADIANS(turn))) AS line,
                *
        FROM    points
        CROSS JOIN
                generate_series(0, 300, 60) AS turn
        )
SELECT  ARRAY_TO_STRING
                (
                ARRAY_AGG
                        (
                        CASE
                                EXISTS
                                (
                                SELECT  NULL
                                FROM    lines
                                WHERE   POINT(x::DOUBLE PRECISION / scale, y::DOUBLE PRECISION / scale) <-> line <= (0.7 / scale)
                                )
                        WHEN    TRUE THEN
                                '#'
                        ELSE    ' '
                        END
                        ORDER BY
                                x
                        ),
                ''
                )
FROM    (
        SELECT  *,
                generate_series(-scale, scale - 1) AS y
        FROM    (
                SELECT  scale, generate_series(-scale, scale - 1) AS x
                FROM    (
                        VALUES
                        (40)
                        ) q (scale)
                ) q
        ) q
GROUP BY
        y
ORDER BY
        y

View query

SELECT  SETSEED(0.201112313);

WITH    RECURSIVE
        params (id, cut, len, alpha, level, spikes) AS
        (
        SELECT  ARRAY[0::INT], 0::DOUBLE PRECISION, 1::DOUBLE PRECISION, 0::DOUBLE PRECISION, 1, FLOOR(RANDOM() * 4)::INT AS spikes
        UNION ALL
        SELECT  id, cut, len, alpha, level, FLOOR(RANDOM() * 4)::INT AS spikes
        FROM    (
                SELECT  id || generate_series(0, spikes) AS id,
                        RANDOM() AS cut, len * RANDOM() * 0.5 AS len, (PI() / 2) * (0.3 + RANDOM() * 0.3) AS alpha, level + 1 AS level
                FROM    params
                ) q
        WHERE   level <= 3
        ),
        tree AS
        (
        SELECT  *, generate_series(0, (1 << (level - 1)) - 1) AS branch
        FROM    params
        ),
        points AS
        (
        SELECT  id,
                0::double precision AS xs,
                0::double precision AS ys,
                len AS xe,
                0::double precision AS ye,
                0::double precision AS alpha,
                branch,
                level,
                spikes
        FROM    tree
        WHERE   id = ARRAY[0]
        UNION ALL
        SELECT  id,
                xs + (xe - xs) * cut AS xs,
                ys + (ye - ys) * cut AS ys,
                xs + (xe - xs) * cut + len * COS(alpha) AS xe,
                ys + (ye - ys) * cut + len * SIN(alpha) AS ye,
                alpha,
                branch,
                level,
                spikes
        FROM    (
                SELECT  t.id, p.id || t.id AS uid, xs, xe, ys, ye, cut, len,
                        p.alpha + t.alpha * ((t.branch % 2) * 2 - 1) AS alpha,
                        t.branch, t.level, t.spikes
                FROM    points p
                JOIN    tree t
                ON      t.id > p.id
                        AND t.id <= (p.id || p.spikes)
                        AND ARRAY_LENGTH(t.id, 1) = p.level + 1
                        AND t.branch BETWEEN p.branch * 2 AND p.branch * 2 + 1
                ) q
        ),
        lines AS
        (
        SELECT  PATH(LSEG(POINT(xs, ys), POINT(xe, ye))::TEXT) * POINT(COS(RADIANS(turn)), SIN(RADIANS(turn))) AS line,
                *
        FROM    points
        CROSS JOIN
                generate_series(0, 300, 60) AS turn
        )
SELECT  ARRAY_TO_STRING
                (
                ARRAY_AGG
                        (
                        CASE
                                EXISTS
                                (
                                SELECT  NULL
                                FROM    lines
                                WHERE   POINT(x::DOUBLE PRECISION / scale, y::DOUBLE PRECISION / scale) <-> line <= (0.7 / scale)
                                )
                        WHEN    TRUE THEN
                                '#'
                        ELSE    ' '
                        END
                        ORDER BY
                                x
                        ),
                ''
                )
FROM    (
        SELECT  *,
                generate_series(-scale, scale - 1) AS y
        FROM    (
                SELECT  scale, generate_series(-scale, scale - 1) AS x
                FROM    (
                        VALUES
                        (40)
                        ) q (scale)
                ) q
        ) q
GROUP BY
        y
ORDER BY
        y

Previous New Year posts:

• Happy New 2010 Year!
• Happy New 2011 Year!

Karen asks:

… I’ve always thought PIVOT and UNPIVOT are signs of a poorly designed database. I mean, is there a legitimate use for them if your model is OK?

I’ve made an actual use for them in a project I’ve been working on for the last several months (which is partly why there were no updates for so long!)

Part of the project is a task management system where each task has several persons related to it. There can be the creator of the task, the person the task is assigned to, the actual author of the task (on behalf of whom the task is created), and the task can be possible completed by a person or deleted by a person. A total of 5 fields related to persons.

Now, we need to take all tasks within a certain time range and list all people involved in them.

Let’s create a sample table and see how would we do that.

Table creation details

SET NOCOUNT ON
GO

DROP TABLE
        [20110630_unpivot].task

DROP SCHEMA
        [20110630_unpivot]
GO

CREATE SCHEMA
        [20110630_unpivot]

CREATE TABLE
        [20110630_unpivot].Task
        (
        id INT NOT NULL PRIMARY KEY IDENTITY,
        ts DATETIME NOT NULL,
        createdBy INT NOT NULL,
        assignedTo INT NOT NULL,
        onBehalfOf INT NOT NULL,
        completedBy INT,
        deletedBy INT,
        stuffing CHAR(1000) NOT NULL DEFAULT ''
        )
GO

CREATE INDEX
        IX_Task_Ts
ON      [20110630_unpivot].Task(ts)
GO

BEGIN TRANSACTION

SELECT  RAND(20110630)

DECLARE @cnt INT

SET @cnt = 0

WHILE @cnt < 1000000
BEGIN
        INSERT
        INTO    [20110630_unpivot].Task
                (
                ts, createdBy, assignedTo, onBehalfOf, completedBy, deletedBy
                )
        VALUES  (
                DATEADD(s, -@cnt, DATEADD(d, 1, '2011-06-30')),
                RAND() * 100000,
                RAND() * 100000,
                RAND() * 100000,
                RAND() * 100000,
                RAND() * 100000
                )
        SET @cnt = @cnt + 1
END

COMMIT

GO

There are 5 persons fields, timestamp and stuffing. The timestamp field is indexed.

Now, let’s find all people involved in the tasks between 2011-06-30 and 2011-06-30 04:00:00. To do this, we could just use 5 queries (each selecting one of the persons) and UNION them:

SELECT  SUM(CAST(id AS BIGINT)), COUNT(*)
FROM    (
        SELECT  createdBy AS id
        FROM    [20110630_unpivot].Task
        WHERE   ts >= '2011-06-30'
                AND ts < '2011-06-30 04:00:00'
        UNION
        SELECT  assignedTo AS id
        FROM    [20110630_unpivot].Task
        WHERE   ts >= '2011-06-30'
                AND ts < '2011-06-30 04:00:00'
        UNION
        SELECT  onBehalfOf AS id
        FROM    [20110630_unpivot].Task
        WHERE   ts >= '2011-06-30'
                AND ts < '2011-06-30 04:00:00'
        UNION
        SELECT  completedBy AS id
        FROM    [20110630_unpivot].Task
        WHERE   ts >= '2011-06-30'
                AND ts < '2011-06-30 04:00:00'
                AND completedBy IS NOT NULL
        UNION
        SELECT  deletedBy AS id
        FROM    [20110630_unpivot].Task
        WHERE   ts >= '2011-06-30'
                AND ts < '2011-06-30 04:00:00'
                AND deletedBy IS NOT NULL
        ) q

[Microsoft][SQL Server Native Client 10.0][SQL Server]Table 'Task'. Scan count 15, logical reads 220765, physical reads 0, read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.
[Microsoft][SQL Server Native Client 10.0][SQL Server]Table 'Worktable'. Scan count 0, logical reads 0, physical reads 0, read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.
[Microsoft][SQL Server Native Client 10.0][SQL Server]Table 'Worktable'. Scan count 0, logical reads 0, physical reads 0, read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.
[Microsoft][SQL Server Native Client 10.0][SQL Server]
 SQL Server Execution Times:
   CPU time = 484 ms,  elapsed time = 410 ms.

  |--Compute Scalar(DEFINE:([Expr1016]=CASE WHEN [globalagg1019]=(0) THEN NULL ELSE [globalagg1021] END, [Expr1017]=CONVERT_IMPLICIT(int,[globalagg1023],0)))
       |--Stream Aggregate(DEFINE:([globalagg1019]=SUM([partialagg1018]), [globalagg1021]=SUM([partialagg1020]), [globalagg1023]=SUM([partialagg1022])))
            |--Parallelism(Gather Streams)
                 |--Compute Scalar(DEFINE:([partialagg1022]=[partialagg1018]))
                      |--Stream Aggregate(DEFINE:([partialagg1018]=Count(*), [partialagg1020]=SUM(CONVERT(bigint,[Union1015],0))))
                           |--Hash Match(Aggregate, HASH:([Union1015]), RESIDUAL:([Union1015] = [Union1015]))
                                |--Concatenation
                                     |--Parallelism(Repartition Streams, Hash Partitioning, PARTITION COLUMNS:([ee].[20110630_unpivot].[Task].[createdBy]))
                                     |    |--Nested Loops(Inner Join, OUTER REFERENCES:([ee].[20110630_unpivot].[Task].[id], [Expr1025]) OPTIMIZED WITH UNORDERED PREFETCH)
                                     |         |--Index Seek(OBJECT:([ee].[20110630_unpivot].[Task].[IX_Task_Ts]), SEEK:([ee].[20110630_unpivot].[Task].[ts] >= '2011-06-30 00:00:00.000' AND [ee].[20110630_unpivot].[Task].[ts] < '2011-06-30 04:00:00.000') ORDERED FORWARD)
                                     |         |--Clustered Index Seek(OBJECT:([ee].[20110630_unpivot].[Task].[PK__Task__3213E83F0B3E4B07]), SEEK:([ee].[20110630_unpivot].[Task].[id]=[ee].[20110630_unpivot].[Task].[id]) LOOKUP ORDERED FORWARD)
                                     |--Parallelism(Repartition Streams, Hash Partitioning, PARTITION COLUMNS:([ee].[20110630_unpivot].[Task].[assignedTo]))
                                     |    |--Nested Loops(Inner Join, OUTER REFERENCES:([ee].[20110630_unpivot].[Task].[id], [Expr1026]) OPTIMIZED WITH UNORDERED PREFETCH)
                                     |         |--Index Seek(OBJECT:([ee].[20110630_unpivot].[Task].[IX_Task_Ts]), SEEK:([ee].[20110630_unpivot].[Task].[ts] >= '2011-06-30 00:00:00.000' AND [ee].[20110630_unpivot].[Task].[ts] < '2011-06-30 04:00:00.000') ORDERED FORWARD)
                                     |         |--Clustered Index Seek(OBJECT:([ee].[20110630_unpivot].[Task].[PK__Task__3213E83F0B3E4B07]), SEEK:([ee].[20110630_unpivot].[Task].[id]=[ee].[20110630_unpivot].[Task].[id]) LOOKUP ORDERED FORWARD)
                                     |--Parallelism(Repartition Streams, Hash Partitioning, PARTITION COLUMNS:([ee].[20110630_unpivot].[Task].[onBehalfOf]))
                                     |    |--Nested Loops(Inner Join, OUTER REFERENCES:([ee].[20110630_unpivot].[Task].[id], [Expr1027]) OPTIMIZED WITH UNORDERED PREFETCH)
                                     |         |--Index Seek(OBJECT:([ee].[20110630_unpivot].[Task].[IX_Task_Ts]), SEEK:([ee].[20110630_unpivot].[Task].[ts] >= '2011-06-30 00:00:00.000' AND [ee].[20110630_unpivot].[Task].[ts] < '2011-06-30 04:00:00.000') ORDERED FORWARD)
                                     |         |--Clustered Index Seek(OBJECT:([ee].[20110630_unpivot].[Task].[PK__Task__3213E83F0B3E4B07]), SEEK:([ee].[20110630_unpivot].[Task].[id]=[ee].[20110630_unpivot].[Task].[id]) LOOKUP ORDERED FORWARD)
                                     |--Parallelism(Repartition Streams, Hash Partitioning, PARTITION COLUMNS:([ee].[20110630_unpivot].[Task].[completedBy]))
                                     |    |--Nested Loops(Inner Join, OUTER REFERENCES:([ee].[20110630_unpivot].[Task].[id], [Expr1028]) OPTIMIZED WITH UNORDERED PREFETCH)
                                     |         |--Index Seek(OBJECT:([ee].[20110630_unpivot].[Task].[IX_Task_Ts]), SEEK:([ee].[20110630_unpivot].[Task].[ts] >= '2011-06-30 00:00:00.000' AND [ee].[20110630_unpivot].[Task].[ts] < '2011-06-30 04:00:00.000') ORDERED FORWARD)
                                     |         |--Clustered Index Seek(OBJECT:([ee].[20110630_unpivot].[Task].[PK__Task__3213E83F0B3E4B07]), SEEK:([ee].[20110630_unpivot].[Task].[id]=[ee].[20110630_unpivot].[Task].[id]),  WHERE:([ee].[20110630_unpivot].[Task].[completedBy] IS NOT NULL) LOOKUP ORDERED FORWARD)
                                     |--Parallelism(Repartition Streams, Hash Partitioning, PARTITION COLUMNS:([ee].[20110630_unpivot].[Task].[deletedBy]))
                                          |--Nested Loops(Inner Join, OUTER REFERENCES:([ee].[20110630_unpivot].[Task].[id], [Expr1029]) OPTIMIZED WITH UNORDERED PREFETCH)
                                               |--Index Seek(OBJECT:([ee].[20110630_unpivot].[Task].[IX_Task_Ts]), SEEK:([ee].[20110630_unpivot].[Task].[ts] >= '2011-06-30 00:00:00.000' AND [ee].[20110630_unpivot].[Task].[ts] < '2011-06-30 04:00:00.000') ORDERED FORWARD)
                                               |--Clustered Index Seek(OBJECT:([ee].[20110630_unpivot].[Task].[PK__Task__3213E83F0B3E4B07]), SEEK:([ee].[20110630_unpivot].[Task].[id]=[ee].[20110630_unpivot].[Task].[id]),  WHERE:([ee].[20110630_unpivot].[Task].[deletedBy] IS NOT NULL) LOOKUP ORDERED FORWARD)

Now we have 51331 persons returned in 410 ms, which required 220765 logical reads.

As we can see in the plan, the table is scanned 5 times (once for each query). This is not very efficient of course. It would be much better if each matching record in the table would be only visited once.

This is where UNPIVOT comes into play.

As its name suggests, it does the reverse of PIVOT, that is moves data from multiple columns to multiple columns. And this is exactly what we need in our case.

Let’s try it:

SELECT  SUM(CAST(personId AS BIGINT)), COUNT(*)
FROM    (
        SELECT  DISTINCT personId
        FROM    [20110630_unpivot].Task
        UNPIVOT
                (
                personId FOR personType IN
                (createdBy, assignedTo, onBehalfOf, completedBy, deletedBy)
                ) p
        WHERE   ts >= '2011-06-30'
                AND ts < '2011-06-30 04:00:00'
        ) q

[Microsoft][SQL Server Native Client 10.0][SQL Server]Table 'Worktable'. Scan count 0, logical reads 0, physical reads 0, read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.
[Microsoft][SQL Server Native Client 10.0][SQL Server]Table 'Task'. Scan count 3, logical reads 44153, physical reads 0, read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.
[Microsoft][SQL Server Native Client 10.0][SQL Server]Table 'Worktable'. Scan count 0, logical reads 0, physical reads 0, read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.
[Microsoft][SQL Server Native Client 10.0][SQL Server]
 SQL Server Execution Times:
   CPU time = 165 ms,  elapsed time = 180 ms.

  |--Compute Scalar(DEFINE:([Expr1013]=CASE WHEN [globalagg1016]=(0) THEN NULL ELSE [globalagg1018] END, [Expr1014]=CONVERT_IMPLICIT(int,[globalagg1020],0)))
       |--Stream Aggregate(DEFINE:([globalagg1016]=SUM([partialagg1015]), [globalagg1018]=SUM([partialagg1017]), [globalagg1020]=SUM([partialagg1019])))
            |--Parallelism(Gather Streams)
                 |--Compute Scalar(DEFINE:([partialagg1019]=[partialagg1015]))
                      |--Stream Aggregate(DEFINE:([partialagg1015]=Count(*), [partialagg1017]=SUM(CONVERT(bigint,[Expr1011],0))))
                           |--Sort(DISTINCT ORDER BY:([Expr1011] ASC))
                                |--Parallelism(Repartition Streams, Hash Partitioning, PARTITION COLUMNS:([Expr1011]))
                                     |--Hash Match(Partial Aggregate, HASH:([Expr1011]), RESIDUAL:([Expr1011] = [Expr1011]))
                                          |--Filter(WHERE:([Expr1011] IS NOT NULL))
                                               |--Nested Loops(Left Outer Join, OUTER REFERENCES:([ee].[20110630_unpivot].[Task].[createdBy], [ee].[20110630_unpivot].[Task].[assignedTo], [ee].[20110630_unpivot].[Task].[onBehalfOf], [ee].[20110630_unpivot].[Task].[completedBy], [ee].[20110630_unpivot].[Task].[deletedBy]))
                                                    |--Nested Loops(Inner Join, OUTER REFERENCES:([ee].[20110630_unpivot].[Task].[id], [Expr1022]) OPTIMIZED WITH UNORDERED PREFETCH)
                                                    |    |--Index Seek(OBJECT:([ee].[20110630_unpivot].[Task].[IX_Task_Ts]), SEEK:([ee].[20110630_unpivot].[Task].[ts] >= '2011-06-30 00:00:00.000' AND [ee].[20110630_unpivot].[Task].[ts] < '2011-06-30 04:00:00.000') ORDERED FORWARD)
                                                    |    |--Clustered Index Seek(OBJECT:([ee].[20110630_unpivot].[Task].[PK__Task__3213E83F14C7B541]), SEEK:([ee].[20110630_unpivot].[Task].[id]=[ee].[20110630_unpivot].[Task].[id]) LOOKUP ORDERED FORWARD)
                                                    |--Constant Scan(VALUES:(([ee].[20110630_unpivot].[Task].[createdBy]),([ee].[20110630_unpivot].[Task].[assignedTo]),([ee].[20110630_unpivot].[Task].[onBehalfOf]),([ee].[20110630_unpivot].[Task].[completedBy]),([ee].[20110630_unpivot].[Task].[deletedBy])))

As we can see, this is more than twice as efficient as a UNION and only takes 44153 page reads to complete.

How does it work internally?

We see in the plan that there is a nested loop between a result of a clustered index seek and something called Constant Scan. The constant scan returns 5 values in each loop and those are the fields listed in the UNPIVOT clause. It just takes each record and outputs fields found there, without rereading the record. This is actually what we wanted.

This behavior can be made more clear if we rewrite the query a little:

SELECT  SUM(CAST(personId AS BIGINT)), COUNT(*)
FROM    (
        SELECT  DISTINCT personId
        FROM    [20110630_unpivot].Task
        CROSS APPLY
                (
                SELECT  createdBy AS personId
                UNION ALL
                SELECT  assignedTo AS personId
                UNION ALL
                SELECT  onBehalfOf AS personId
                UNION ALL
                SELECT  completedBy AS personId
                UNION ALL
                SELECT  deletedBy AS personId
                ) p
        WHERE   ts >= '2011-06-30'
                AND ts < '2011-06-30 04:00:00'
                AND personId IS NOT NULL
        ) q

[Microsoft][SQL Server Native Client 10.0][SQL Server]Table 'Worktable'. Scan count 0, logical reads 0, physical reads 0, read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.
[Microsoft][SQL Server Native Client 10.0][SQL Server]Table 'Task'. Scan count 3, logical reads 44153, physical reads 0, read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.
[Microsoft][SQL Server Native Client 10.0][SQL Server]Table 'Worktable'. Scan count 0, logical reads 0, physical reads 0, read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.
[Microsoft][SQL Server Native Client 10.0][SQL Server]
 SQL Server Execution Times:
   CPU time = 149 ms,  elapsed time = 176 ms.

  |--Compute Scalar(DEFINE:([Expr1004]=CASE WHEN [globalagg1007]=(0) THEN NULL ELSE [globalagg1009] END, [Expr1005]=CONVERT_IMPLICIT(int,[globalagg1011],0)))
       |--Stream Aggregate(DEFINE:([globalagg1007]=SUM([partialagg1006]), [globalagg1009]=SUM([partialagg1008]), [globalagg1011]=SUM([partialagg1010])))
            |--Parallelism(Gather Streams)
                 |--Compute Scalar(DEFINE:([partialagg1010]=[partialagg1006]))
                      |--Stream Aggregate(DEFINE:([partialagg1006]=Count(*), [partialagg1008]=SUM(CONVERT(bigint,[Union1003],0))))
                           |--Sort(DISTINCT ORDER BY:([Union1003] ASC))
                                |--Parallelism(Repartition Streams, Hash Partitioning, PARTITION COLUMNS:([Union1003]))
                                     |--Hash Match(Partial Aggregate, HASH:([Union1003]), RESIDUAL:([Union1003] = [Union1003]))
                                          |--Filter(WHERE:([Union1003] IS NOT NULL))
                                               |--Nested Loops(Inner Join, OUTER REFERENCES:([ee].[20110630_unpivot].[Task].[createdBy], [ee].[20110630_unpivot].[Task].[assignedTo], [ee].[20110630_unpivot].[Task].[onBehalfOf], [ee].[20110630_unpivot].[Task].[completedBy], [ee].[20110630_unpivot].[Task].[deletedBy]))
                                                    |--Nested Loops(Inner Join, OUTER REFERENCES:([ee].[20110630_unpivot].[Task].[id], [Expr1013]) OPTIMIZED WITH UNORDERED PREFETCH)
                                                    |    |--Index Seek(OBJECT:([ee].[20110630_unpivot].[Task].[IX_Task_Ts]), SEEK:([ee].[20110630_unpivot].[Task].[ts] >= '2011-06-30 00:00:00.000' AND [ee].[20110630_unpivot].[Task].[ts] < '2011-06-30 04:00:00.000') ORDERED FORWARD)
                                                    |    |--Clustered Index Seek(OBJECT:([ee].[20110630_unpivot].[Task].[PK__Task__3213E83F14C7B541]), SEEK:([ee].[20110630_unpivot].[Task].[id]=[ee].[20110630_unpivot].[Task].[id]) LOOKUP ORDERED FORWARD)
                                                    |--Constant Scan(VALUES:(([ee].[20110630_unpivot].[Task].[createdBy]),([ee].[20110630_unpivot].[Task].[assignedTo]),([ee].[20110630_unpivot].[Task].[onBehalfOf]),([ee].[20110630_unpivot].[Task].[completedBy]),([ee].[20110630_unpivot].[Task].[deletedBy])))

Here, we just take each record and explode it into 5 records using CROSS APPLY.

This yields exactly same plan, exactly same I/O and exactly same output. In fact, that’s exactly what the UNPIVOT query does.

Hope that helps.

I’m always glad to answer the questions regarding database queries.

Ask me a question

While SQL is more or less standardized, the optimizers are implemented differently in the different systems. Some systems cannot join tables with anything other than nested loops, the other can only GROUP BY using a sort, etc.

So when you write a join in, say, MySQL, you cannot expect it to be a sort merge join (and you should consider this fact when designing the query). Or, when you write a DISTINCT in SQL Server, you can’t expect a loose index scan. These are limitations put by their optimizers.

However, in the last three months I noticed a great improvement in the queries where I could not expect any.

It started when I tried to debug this in SQL Server:

SELECT  DISTINCT order
FROM    orderItem

while yielded this plan:

Similar results were obtained on Oracle:

Plan hash value: 1345318323

---------------------------------------------------------------------------------------------------------------------------------
| Id  | Operation                                                         | Name        | Rows  | Bytes | Cost (%CPU)| Time     |
---------------------------------------------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT                                                  |             |       |   200 |     2  (50)| 00:00:01 |
|   1 |  REMOTE SPANC QUERY (SQLSERVER, MYSQL, POSTGRESQL, STACKOVERFLOW) |             |       |   200 |     2  (50)| 00:00:01 |
---------------------------------------------------------------------------------------------------------------------------------

, MySQL:

+----+-------------+-----------+-------+---------------+---------+---------+------+---------+-----------------------------------------------------+
| id | select_type | table     | type  | possible_keys | key     | key_len | ref  | rows    | Extra                                               |
+----+-------------+-----------+-------+---------------+---------+---------+------+---------+-----------------------------------------------------+
|  1 | SIMPLE      | orderItem | spanc | NULL          | ALL     | NULL    | NULL |         | Using Oracle, PostgreSQL, SQL Server, StackOverflow |
+----+-------------+-----------+-------+---------------+---------+---------+------+---------+-----------------------------------------------------+

and PostgreSQL:

Seq Scan on OrderItem  (cost=0.00..6.44 width=4)
 -> Remote Scan on SPANC (Oracle, MySQL, SQL Server, StackOverflow)   (cost=0.00..100.00 width=4)

Network analysis has shown weird encrypted activity between the servers in my internal network which host SQL Server, Oracle, PostgreSQL and MySQL servers.

Ultimately, there was unencrypted activity outside of the internal network which turned out to be an HTTP POST request followed by several GET polls to http://stackoverflow.com/questions/5518080/distinct-optimization.

It seems that the developers of major database systems agreed to share the knowledge about the most efficient query plans in some kind of a distributed storage (which probably is called SPANC as we can see in the query plans) and provide an interface to access each other’s systems.

It also seems that these systems treat Stack Overflow as an external optimization engine where the most experienced developers can build their plans for them in a most efficient way.

I would be glad to have further clarification from the companies staff.

This also begs a question: how many of regular Stack Overflow participants are in fact query engines disguised as curious fellow developers?

It would be definitely nice to know.

I have a query where I have a custom developed UDF that is used to calculate whether or not certain points are within a polygon (first query in UNION) or circular (second query in UNION) shape.

SELECT  a.geo_boundary_id, …
FROM     geo_boundary_vertex a, …
…
GROUP BY
        a.geo_boundary_id
UNION
SELECT  b.geo_boundary_id, …
FROM     geo_boundary b, …
…
GROUP BY
        b.geo_boundary_id

When I run an explain for the query I get filesort for both queries within the UNION.

Now, I can split the queries up and use the ORDER BY NULL trick to get rid of the filesort however when I attempt to add that to the end of a UNION it doesn’t work.

How do I get rid of the filesort?

In MySQL, GROUP BY also implies ORDER BY on the same set of expressions in the same order. That’s why it adds an additional filesort operation to sort the resultset if it does not come out naturally sorted (say, from an index).

This is not always a desired behavior, and MySQL manual suggests adding ORDER BY NULL to the queries where sorting is not required. This can improve performance of the queries.

Let’s create a sample table and see:

Table creation details

CREATE TABLE filler (
        id INT NOT NULL PRIMARY KEY AUTO_INCREMENT
) ENGINE=Memory;

CREATE TABLE grouping (
        id INT NOT NULL PRIMARY KEY AUTO_INCREMENT,
        value1 INT NOT NULL,
        value2 INT NOT NULL
) ENGINE=InnoDB;

DELIMITER $$

CREATE PROCEDURE prc_filler(cnt INT)
BEGIN
        DECLARE _cnt INT;
        SET _cnt = 1;
        WHILE _cnt <= cnt DO
                INSERT
                INTO    filler
                SELECT  _cnt;
                SET _cnt = _cnt + 1;
        END WHILE;
END
$$

DELIMITER ;

START TRANSACTION;
CALL prc_filler(100000);
COMMIT;

INSERT
INTO    grouping (value1, value2)
SELECT  CEILING(RAND(20110330) * 300000),
        CEILING(RAND(20110330 << 1) * 300000)
FROM    filler
CROSS JOIN
        (
        SELECT  id
        FROM    filler
        LIMIT 30
        ) q;

The table contains 3,000,000 random records with value1 and value2 between 1 and 300,000.

Here’s the plan we get with a mere UNION of two GROUP BY queries:

SELECT  value1 AS value
FROM    grouping
GROUP BY
        value1
UNION
SELECT  value2 AS value
FROM    grouping
GROUP BY
        value2
LIMIT 10

select `20110330_group`.`grouping`.`value1` AS `value` from `20110330_group`.`grouping` group by `20110330_group`.`grouping`.`value1` union select `20110330_group`.`grouping`.`value2` AS `value` from `20110330_group`.`grouping` group by `20110330_group`.`grouping`.`value2` limit 10

We see that there is a filesort in each of the queries.

MySQL does allow using ORDER BY in the queries merged with UNION or UNION ALL. To do this, we just need to wrap each query into a set of parentheses:

(
SELECT  value1 AS value
FROM    grouping
GROUP BY
        value1
ORDER BY
        NULL
)
UNION
(
SELECT  value2 AS value
FROM    grouping
GROUP BY
        value2
ORDER BY
        NULL
)
LIMIT 10

(select `20110330_group`.`grouping`.`value1` AS `value` from `20110330_group`.`grouping` group by `20110330_group`.`grouping`.`value1` order by NULL) union (select `20110330_group`.`grouping`.`value2` AS `value` from `20110330_group`.`grouping` group by `20110330_group`.`grouping`.`value2` order by NULL) limit 10

However, the plan remained the same. Why?

MySQL‘s documentation says:

Use of ORDER BY for individual SELECT statements implies nothing about the order in which the rows appear in the final result because UNION by default produces an unordered set of rows. Therefore, the use of ORDER BY in this context is typically in conjunction with LIMIT, so that it is used to determine the subset of the selected rows to retrieve for the SELECT, even though it does not necessarily affect the order of those rows in the final UNION result. If ORDER BY appears without LIMIT in a SELECT, it is optimized away because it will have no effect anyway.

This means that the optimizer just removes ORDER BY from the UNION parts if they are not used along with LIMIT.

This is of course a good idea: since individual ORDER BY have no effect on the order of the final query anyway, there is no use in executing them or even taking them into account.

However, this idea would be much better if the same were also true for GROUP BY. Currently, the optimizer does not optimize away the ordering behavior of the GROUP BY queries which are parts of a UNION and they cannot be cured with ORDER BY NULL (whose only goal is not to order) since this is removed by the optimizer.

However, since only ORDER BY clauses not accompanied with a LIMIT are thrown away, we could just add a LIMIT. Of course it should be large enough to guarantee that all record would be returned.

Let’s see:

(
SELECT  value1 AS value
FROM    grouping
GROUP BY
        value1
ORDER BY
        NULL
LIMIT 10000000000
)
UNION
(
SELECT  value2 AS value
FROM    grouping
GROUP BY
        value2
ORDER BY
        NULL
LIMIT 10000000000
)
LIMIT 10

(select `20110330_group`.`grouping`.`value1` AS `value` from `20110330_group`.`grouping` group by `20110330_group`.`grouping`.`value1` order by NULL limit 10000000000) union (select `20110330_group`.`grouping`.`value2` AS `value` from `20110330_group`.`grouping` group by `20110330_group`.`grouping`.`value2` order by NULL limit 10000000000) limit 10

Now, there are no filesort operations in the plan and the query runs 20% faster.

This is not a very elegant solution of course. More than that, a solution similar to it was used for SQL Server 2000 which does not allow using ORDER BY without a TOP in the inline views. In SQL Server 2000, TOP 100% forced the order of the nested queries and usually made them spooled or otherwise materialized.

It was quite a nasty surprise when SQL Server 2005 has improved its optimizer to detect such tricks and ignore ORDER BY for TOP 100% queries. However, with all improvements introduced in SQL Server 2005, most of these queries could be just rewritten in a more clean and efficient way.

Nevertheless, this solution is still safe to use, because it does not change the semantics of the query (if LIMIT is chosen large enough), but is just an optimizer hack. In the worst case, the query will just become as slow as it initially was, and an extra filesort is not the worst of all things that can happen to a query.

Meanwhile, I’ve posted it as bug 60702 to MySQL bug database and am hoping they’ll fix it in the next release.

Victor asks:

I have a table which I will call sale to protect the innocent:

Sale

id	product	price	amount	date

I need to retrieve ultimate values of price, amount and date for each product:

SELECT  product,
        MIN(price), MAX(price),
        MIN(amount), MAX(amount),
        MIN(date), MAX(date)
FROM    sale
GROUP BY
        product

The query only returns about 100 records.

I have all these fields indexed (together with product), but this still produces a full table scan over 3,000,000 records.

How do I speed up this query?

To retrieve the ultimate values of the fields, MySQL would just need to make a loose index scan over each index and take the max and min value of the field for each product.

However, the optimizer won’t do when multiple indexes are involved. Instead, it will revert to a full scan.

There is a workaround for it. Let’s create a sample table and see them:

Table creation details

CREATE TABLE filler (
        id INT NOT NULL PRIMARY KEY AUTO_INCREMENT
) ENGINE=Memory;

CREATE TABLE sale
        (
        id INT NOT NULL PRIMARY KEY AUTO_INCREMENT,
        product INT NOT NULL,
        amount INT NOT NULL,
        price NUMERIC(20, 2) NOT NULL,
        dt DATETIME NOT NULL,
        KEY ix_sale_product_amount (product, amount),
        KEY ix_sale_product_price (product, price),
        KEY ix_sale_product_dt (product, dt)
        )
ENGINE=InnoDB;

DELIMITER $$

CREATE PROCEDURE prc_filler(cnt INT)
BEGIN
        DECLARE _cnt INT;
        SET _cnt = 1;
        WHILE _cnt <= cnt DO
                INSERT
                INTO    filler
                SELECT  _cnt;
                SET _cnt = _cnt + 1;
        END WHILE;
END
$$

DELIMITER ;

START TRANSACTION;
CALL prc_filler(50000);
COMMIT;

INSERT
INTO    sale (product, amount, price, dt)
SELECT  CEILING(RAND(20110227) * 100),
        CEILING(RAND(20110227 << 1) * 1000) + 100,
        CEILING(RAND(20110227 << 1 + 1) * 10000) / 100.00 + 20.00,
        '2011-03-27' - INTERVAL CEILING(RAND(20110227 << 1 + 2) * 10000000) SECOND
FROM    filler
CROSS JOIN
        (
        SELECT  id
        FROM    filler
        ORDER BY
                id
        LIMIT 60
        ) q;

This table contains 3M random records for 100 distinct products and is indexed appropriately.

Here’s a straightforward query:

SELECT  product,
        MIN(price), MAX(price),
        MIN(amount), MAX(amount),
        MIN(dt), MAX(dt)
FROM    sale
GROUP BY
        product

select `20110327_split`.`sale`.`product` AS `product`,min(`20110327_split`.`sale`.`price`) AS `MIN(price)`,max(`20110327_split`.`sale`.`price`) AS `MAX(price)`,min(`20110327_split`.`sale`.`amount`) AS `MIN(amount)`,max(`20110327_split`.`sale`.`amount`) AS `MAX(amount)`,min(`20110327_split`.`sale`.`dt`) AS `MIN(dt)`,max(`20110327_split`.`sale`.`dt`) AS `MAX(dt)` from `20110327_split`.`sale` group by `20110327_split`.`sale`.`product`

As was expected, the query uses a full table scan (or, rather, an index scan) and takes almost 14 seconds to complete.

In order to make the engine use three separate indexes, we would need to split the query into three queries each searching for ultimate values on one column, and then combine the results.

Each of these queries would be a MIN / MAX query on a trailing column of a composite index, combined with a GROUP BY on the index’s leading columns, and, as such, would be subject to loose index scan optimization.

To combine the results, we will of course just join them on product. This will be quite efficient too since the resultsets are going to be quite small (100 records each, exactly).

And here is the query:

SELECT  *
FROM    (
        SELECT  product, MIN(amount), MAX(amount)
        FROM    sale
        GROUP BY
                product
        ) qa
NATURAL JOIN
        (
        SELECT  product, MIN(price), MAX(price)
        FROM    sale
        GROUP BY
                product
        ) qp
NATURAL JOIN
        (
        SELECT  product, MIN(dt), MAX(dt)
        FROM    sale
        GROUP BY
                product
        ) qd

select `qa`.`product` AS `product`,`qa`.`MIN(amount)` AS `MIN(amount)`,`qa`.`MAX(amount)` AS `MAX(amount)`,`qp`.`MIN(price)` AS `MIN(price)`,`qp`.`MAX(price)` AS `MAX(price)`,`qd`.`MIN(dt)` AS `MIN(dt)`,`qd`.`MAX(dt)` AS `MAX(dt)` from (select `20110327_split`.`sale`.`product` AS `product`,min(`20110327_split`.`sale`.`amount`) AS `MIN(amount)`,max(`20110327_split`.`sale`.`amount`) AS `MAX(amount)` from `20110327_split`.`sale` group by `20110327_split`.`sale`.`product`) `qa` join (select `20110327_split`.`sale`.`product` AS `product`,min(`20110327_split`.`sale`.`price`) AS `MIN(price)`,max(`20110327_split`.`sale`.`price`) AS `MAX(price)` from `20110327_split`.`sale` group by `20110327_split`.`sale`.`product`) `qp` join (select `20110327_split`.`sale`.`product` AS `product`,min(`20110327_split`.`sale`.`dt`) AS `MIN(dt)`,max(`20110327_split`.`sale`.`dt`) AS `MAX(dt)` from `20110327_split`.`sale` group by `20110327_split`.`sale`.`product`) `qd` where ((`qp`.`product` = `qa`.`product`) and (`qd`.`product` = `qa`.`product`))

Each individual query is executed with using index for group-by, that is jumping over the min and max values in a loose index scan.

The queries are combined in a join which, despite not using the indexes, is still very fast because only 100 records are joined (of course fitting into a join buffer).

The overall query time is only 10 ms.

Hope that helps.

I’m always glad to answer the questions regarding database queries.

Ask me a question

I believe this could be best illustrated with an example:

The records are ordered by id, and within this order there are continuous series of records which share the same value of source. In the table above, the series are separated by thick lines.

We want to calculate some aggregates across each of the series: MIN, MAX, SUM, AVG, whatever:

This can be used for different things. I used that for:

• Reading sensors from a moving elevator (thus tracking its position)
• Recording user’s activity on a site
• Tracking the primary node in a server cluster

, but almost any seasoned database developer can recall a need for such a query.

As you can see, the values of source are repeating so a mere GROUP BY won’t work here.

In the systems supporting window functions there is a workaround for that:

SELECT  source, MIN(value), MAX(value), SUM(value), AVG(value)::NUMERIC(20, 2)
FROM    (
        SELECT  *,
                ROW_NUMBER() OVER (PARTITION BY source ORDER BY id) AS rno,
                ROW_NUMBER() OVER (ORDER BY id) AS rne
        FROM    (
                VALUES
                (1, 1, 10),
                (2, 1, 20),
                (3, 2, 15),
                (4, 2, 25),
                (5, 1, 45),
                (6, 3, 50),
                (7, 3, 35),
                (8, 1, 40),
                (9, 1, 10)
                ) series (id, source, value)
        ) q
GROUP BY
        source, rne - rno
ORDER BY
        MIN(id)

Sort  (cost=1.40..1.42 rows=9 width=28)
  Sort Key: (min(q.id))
  ->  HashAggregate  (cost=1.01..1.25 rows=9 width=28)
        ->  Subquery Scan q  (cost=0.58..0.85 rows=9 width=28)
              ->  WindowAgg  (cost=0.58..0.74 rows=9 width=12)
                    ->  Sort  (cost=0.58..0.60 rows=9 width=12)
                          Sort Key: "*VALUES*".column1
                          ->  WindowAgg  (cost=0.26..0.44 rows=9 width=12)
                                ->  Sort  (cost=0.26..0.28 rows=9 width=12)
                                      Sort Key: "*VALUES*".column2, "*VALUES*".column1
                                      ->  Values Scan on "*VALUES*"  (cost=0.00..0.11 rows=9 width=12)

(this is PostgreSQL).

The idea behind this solution is that the overall order of id is retained within each of the series but is broken whenever the series break. Thus, a difference between the overall row number and the source-wise row number is an invariant within each series and is guaranteed to change as the series break. Hence, we can GROUP BY on it (along with the source).

This workaround is nice and elegant, however it requires three sorts and one hash aggregate (apart from the support for window functions of course).

SERIES function

My proposal is to implement a certain extension to the GROUP BY clause, namely:

GROUP BY SERIES (series_expression) OVER ([PARTITION BY partitioning_expression] ORDER BY ordering_expression)

The logic is quite simple: it would find continuous blocks of series_expression as if the records were ordered by series_ordering_expression and assign them to the same group. Within this group, all standard aggregation functions could be supported.

The expression:

SERIES (series_expression) OVER ([PARTITION BY partitioning_expression] ORDER BY ordering_expression)

could also serve as a normal window function in the systems that support it. In this case, it would return the ordinal number of the series within each partition.

Implementation

Its implementation would be quite simple: the records sharing the same value of partitioning_expression should be ordered on ordering_expression, a zero-based (or one-based) counter a variable holding the previous value of the series_expression should be set up. Whenever the value of the series_expression changes, the counter is incremented (and returned as an output of SERIES) and the variable gets the new value of series_expression.

Using the session variables, this can be easily emulated in MySQL:

SELECT  source,
        MIN(value) AS min,
        MAX(value) AS max,
        SUM(value) AS sum,
        CAST(AVG(value) AS DECIMAL(20, 2)) AS avg
FROM    (
        SELECT  @series := @series + (COALESCE(@source <> source, 0)) AS series,
                @source := source AS newsource,
                q.*
        FROM    (
                SELECT  @series := 1, @source := NULL
                ) vars
        STRAIGHT_JOIN
                (
                SELECT  1 AS id, 1 AS source, 10 AS value
                UNION ALL
                SELECT  2 AS id, 1 AS source, 20 AS value
                UNION ALL
                SELECT  3 AS id, 2 AS source, 15 AS value
                UNION ALL
                SELECT  4 AS id, 2 AS source, 25 AS value
                UNION ALL
                SELECT  5 AS id, 1 AS source, 45 AS value
                UNION ALL
                SELECT  6 AS id, 3 AS source, 50 AS value
                UNION ALL
                SELECT  7 AS id, 3 AS source, 35 AS value
                UNION ALL
                SELECT  8 AS id, 1 AS source, 40 AS value
                UNION ALL
                SELECT  9 AS id, 1 AS source, 10 AS value
                ) q
        ORDER BY
                id
        ) q
GROUP BY
        series

select `q`.`source` AS `source`,min(`q`.`value`) AS `min`,max(`q`.`value`) AS `max`,sum(`q`.`value`) AS `sum`,cast(avg(`q`.`value`) as decimal(20,2)) AS `avg` from (select (@series:=((@series) + coalesce(((@source) <> `q`.`source`),0))) AS `series`,(@source:=`q`.`source`) AS `newsource`,`q`.`id` AS `id`,`q`.`source` AS `source`,`q`.`value` AS `value` from (select (@series:=1) AS `@series := 1`,(@source:=NULL) AS `@source := NULL`) `vars` straight_join (select 1 AS `id`,1 AS `source`,10 AS `value` union all select 2 AS `id`,1 AS `source`,20 AS `value` union all select 3 AS `id`,2 AS `source`,15 AS `value` union all select 4 AS `id`,2 AS `source`,25 AS `value` union all select 5 AS `id`,1 AS `source`,45 AS `value` union all select 6 AS `id`,3 AS `source`,50 AS `value` union all select 7 AS `id`,3 AS `source`,35 AS `value` union all select 8 AS `id`,1 AS `source`,40 AS `value` union all select 9 AS `id`,1 AS `source`,10 AS `value`) `q` order by `q`.`id`) `q` group by `q`.`series`

, with only one filesort (which could be avoided if the values were ordered with an index).

Grouping on expressions sharing the order

This clause would also allow running certain grouping queries more efficiently.

Say, we need to build a yearly report for sales on a web site. Let’s create a sample table:

Table creation details

CREATE TABLE filler (
        id INT NOT NULL PRIMARY KEY AUTO_INCREMENT
) ENGINE=Memory;

CREATE TABLE sales (
        id INT NOT NULL,
        dt DATETIME NOT NULL,
        amount DECIMAL(20, 2) NOT NULL
) ENGINE=InnoDB;

CREATE INDEX
        ix_sales_dt_amount
ON      sales (dt, amount);

DELIMITER $$

CREATE PROCEDURE prc_filler(cnt INT)
BEGIN
        DECLARE _cnt INT;
        SET _cnt = 1;
        WHILE _cnt <= cnt DO
                INSERT
                INTO    filler
                SELECT  _cnt;
                SET _cnt = _cnt + 1;
        END WHILE;
END
$$

DELIMITER ;

START TRANSACTION;
CALL prc_filler(500000);
COMMIT;

INSERT
INTO    sales
SELECT  id,
        CAST('2011-02-18' AS DATE) - INTERVAL id * 10 MINUTE,
        (10000 + CEILING(RAND(20110218) * 10000)) / 100
FROM    filler;

Normally, we do it with this query:

SELECT  YEAR(dt), SUM(amount)
FROM    sales
WHERE   dt >= '2005-01-01'
GROUP BY
        YEAR(dt)
ORDER BY
        NULL

select year(`20110218_series`.`sales`.`dt`) AS `YEAR(dt)`,sum(`20110218_series`.`sales`.`amount`) AS `SUM(amount)` from `20110218_series`.`sales` where (`20110218_series`.`sales`.`dt` >= '2005-01-01') group by year(`20110218_series`.`sales`.`dt`) order by NULL

The query is pretty fast, but we see a using temporary in the plan. Since the order of of dt is always that of YEAR(dt), and the records are selected from the index on dt (which naturally come ordered), the temporary table is redundant. A sort grouping can be used instead.

We have no means to tell MySQL (or any other database system) to tell that the orders match, but with the extension proposed, we could do just that:

SELECT  YEAR(MIN(dt)), SUM(amount)
FROM    sales
WHERE   dt >= '2005-01-01'
GROUP BY
        SERIES(YEAR(dt)) OVER (ORDER BY dt)

Since YEAR(dt) will always form continuous series being ordered by dt, this is the same as the query above, but filesort could be avoided.

Let’s emulate this using session variables:

SELECT  CAST(year AS UNSIGNED), CAST(sum_amount AS DECIMAL(20, 2))
FROM    (
        SELECT  @sum := 0,
                @year := YEAR(MIN(dt))
        FROM    sales
        WHERE   dt >= '2005-01-01'
        ) vars
STRAIGHT_JOIN
        (
        SELECT  @sum AS sum_amount,
                @sum := CASE WHEN @year = YEAR(dt) THEN @sum + amount ELSE amount END,
                @year = YEAR(dt) AS series,
                @year AS year,
                @year := YEAR(dt)
        FROM    sales
        WHERE   dt >= '2005-01-01'
        ORDER BY
                dt
        ) q
WHERE   series = 0
UNION ALL
SELECT  CAST(@year AS UNSIGNED), CAST(@sum AS DECIMAL(20, 2))

select cast(`q`.`year` as unsigned) AS `CAST(year AS UNSIGNED)`,cast(`q`.`sum_amount` as decimal(20,2)) AS `CAST(sum_amount AS DECIMAL(20, 2))` from (select (@sum:=0) AS `@sum := 0`,(@year:=year(min(`20110218_series`.`sales`.`dt`))) AS `@year := YEAR(MIN(dt))` from `20110218_series`.`sales` where (`20110218_series`.`sales`.`dt` >= '2005-01-01')) `vars` straight_join (select (@sum) AS `sum_amount`,(@sum:=(case when ((@year) = year(`20110218_series`.`sales`.`dt`)) then ((@sum) + `20110218_series`.`sales`.`amount`) else `20110218_series`.`sales`.`amount` end)) AS `@sum := CASE WHEN @year = YEAR(dt) THEN @sum + amount ELSE amount END`,((@year) = year(`20110218_series`.`sales`.`dt`)) AS `series`,(@year) AS `year`,(@year:=year(`20110218_series`.`sales`.`dt`)) AS `@year := YEAR(dt)` from `20110218_series`.`sales` where (`20110218_series`.`sales`.`dt` >= '2005-01-01') order by `20110218_series`.`sales`.`dt`) `q` where (`q`.`series` = 0) union all select cast((@year) as unsigned) AS `CAST(@year AS UNSIGNED)`,cast((@sum) as decimal(20,2)) AS `CAST(@sum AS DECIMAL(20, 2))`

We see that though inline view support is pretty inefficient in MySQL and the resulting query is not as fast as it should be, there is no using temporary and the query is more efficient. With a native support for SERIES(), it would be yet more fast.

Conclusion

A window function (or an extension for GROUP BY clause) that would allow grouping continuous series of an expression in an ordered dataset would ease writing queries that need to aggregate such series, make them more efficient and even improve certain kinds of grouping queries where the grouping expression shares its order with one of the indexes.

Update of Feb 21 2011

Another nice solution proposed by @delostilos:

SELECT  MIN(source), series, SUM(value)
FROM    (
        SELECT  *,
                SUM(COALESCE((source <> ns)::INTEGER, 0)) OVER (ORDER BY id) AS series
        FROM    (
                SELECT  series.*,
                        LAG(source) OVER (ORDER BY id) AS ns
                FROM    (
                        VALUES
                        (1, 1, 10),
                        (2, 1, 20),
                        (3, 2, 15),
                        (4, 2, 25),
                        (5, 1, 45),
                        (6, 3, 50),
                        (7, 3, 35),
                        (8, 1, 40),
                        (9, 1, 10)
                        ) series (id, source, value)
                ) q
        ) q
GROUP BY
        series

HashAggregate  (cost=0.84..0.98 rows=9 width=16)
  ->  WindowAgg  (cost=0.26..0.68 rows=9 width=16)
        ->  WindowAgg  (cost=0.26..0.41 rows=9 width=12)
              ->  Sort  (cost=0.26..0.28 rows=9 width=12)
                    Sort Key: "*VALUES*".column1
                    ->  Values Scan on "*VALUES*"  (cost=0.00..0.11 rows=9 width=12)

This is more efficient than double ROW_NUMBER, since the engine only needs to sort on the id expression at most once (or does not need to sort at all if it comes sorted from the previous recordset or from the index).

However, the final GROUP BY still cannot use sort aggregation without additional sorting, since the output of the analytic SUM() cannot be assumed to have the same order as id. That’s why the native support from the system would still be an improvement.

Aryé asks:

Thanks for your article about late row lookups in MySQL.

I have two questions for you please:

• Is this workaround specific to MyISAM engine?
• How does PostgreSQL handle this?

The questions concerns a certain workaround for MySQL LIMIT … OFFSET queries like this:

SELECT  *
FROM    mytable
ORDER BY
        id
LIMIT   10
OFFSET  10000

which can be improved using a little rewrite:

SELECT  m.*
FROM    (
        SELECT  id
        FROM    mytable
        ORDER BY
                id
        LIMIT   10
        OFFSET  10000
        ) q
JOIN    mytable m
ON      m.id = q.id
ORDER BY
        m.id

For the rationale behind this improvement, please read the original article.

Now, to the questions.

The second questions is easy: PostgreSQL won’t pull the fields from the table until it really needs them. If a query involving an ORDER BY along with LIMIT and OFFSET is optimized to use the index for the ORDER BY part, the table lookups won’t happen for the records skipped.

Though PostgreSQL does not reflect the table lookups in the EXPLAIN output, a simple test would show us that they are done only LIMIT times, not OFFSET + LIMIT (like MySQL does).

Now, let’s try to answer the first question: will this trick improve the queries against an InnoDB table?

To do this, we will create a sample table:

Table creation details

CREATE TABLE filler (
        id INT NOT NULL PRIMARY KEY AUTO_INCREMENT
) ENGINE=Memory;

CREATE TABLE lookup (
        id INT NOT NULL PRIMARY KEY,
        value INT NOT NULL,
        shorttxt TEXT NOT NULL,
        longtxt TEXT NOT NULL
) ENGINE=InnoDB ROW_FORMAT=COMPACT;

CREATE INDEX
        ix_lookup_value
ON      lookup (value);

DELIMITER $$

CREATE PROCEDURE prc_filler(cnt INT)
BEGIN
        DECLARE _cnt INT;
        SET _cnt = 1;
        WHILE _cnt <= cnt DO
                INSERT
                INTO    filler
                SELECT  _cnt;
                SET _cnt = _cnt + 1;
        END WHILE;
END
$$

DELIMITER ;

START TRANSACTION;
CALL prc_filler(100000);
COMMIT;

INSERT
INTO    lookup
SELECT  id, CEILING(RAND(20110211) * 1000000),
        RPAD('', CEILING(RAND(20110211 << 1) * 100), '*'),
        RPAD('', CEILING(8192 + RAND(20110211 << 1) * 100), '*')
FROM    filler;

There is one indexed INT column and two TEXT columns, shorttxt storing short strings (1 to 100 characters), longtxt storing long strings (8193 to 8293 characters).

Let’s run some queries against this table.

PRIMARY KEY and the INT column

No rewrite:

SELECT  value
FROM    lookup
ORDER BY
        id
LIMIT   10
OFFSET  90000

Query results

select `20110211_late`.`lookup`.`value` AS `value` from `20110211_late`.`lookup` order by `20110211_late`.`lookup`.`id` limit 90000,10

Rewrite:

Query results

SELECT  l.value
FROM    (
        SELECT  id
        FROM    lookup
        ORDER BY
                id
        LIMIT   10
        OFFSET  90000
        ) q
JOIN    lookup l
ON      l.id = q.id
ORDER BY
        q.id

select `20110211_late`.`l`.`value` AS `value` from (select `20110211_late`.`lookup`.`id` AS `id` from `20110211_late`.`lookup` order by `20110211_late`.`lookup`.`id` limit 90000,10) `q` join `20110211_late`.`lookup` `l` where (`20110211_late`.`l`.`id` = `q`.`id`) order by `q`.`id`

As you can see, there is almost no difference (41 ms vs 40 ms).

InnoDB tables are clustered on their PRIMARY KEY columns, which means the the index on id (used to serve the ORDER BY condition) contains all the data the query needs. There is a negligible benefit from not lookup up the value columns at the index pages because the column is tiny and the index pages need to be read anyway.

PRIMARY KEY and the short TEXT column

Rewrite:

SELECT  LENGTH(l.shorttxt)
FROM    (
        SELECT  id
        FROM    lookup
        ORDER BY
                id
        LIMIT   10
        OFFSET  90000
        ) q
JOIN    lookup l
ON      l.id = q.id
ORDER BY
        q.id

Query results

select length(`20110211_late`.`l`.`shorttxt`) AS `LENGTH(l.shorttxt)` from (select `20110211_late`.`lookup`.`id` AS `id` from `20110211_late`.`lookup` order by `20110211_late`.`lookup`.`id` limit 90000,10) `q` join `20110211_late`.`lookup` `l` where (`20110211_late`.`l`.`id` = `q`.`id`) order by `q`.`id`

No rewrite:

SELECT  LENGTH(shorttxt)
FROM    lookup
ORDER BY
        id
LIMIT   10
OFFSET  90000

Query results

select length(`20110211_late`.`lookup`.`shorttxt`) AS `LENGTH(shorttxt)` from `20110211_late`.`lookup` order by `20110211_late`.`lookup`.`id` limit 90000,10

There is quite a significant difference (92 ms vs 40 ms) the reasons for which we will discuss a little bit later, after we see the results of the third query.

PRIMARY KEY and the long TEXT column

Rewrite:

SELECT  LENGTH(l.longtxt)
FROM    (
        SELECT  id
        FROM    lookup
        ORDER BY
                id
        LIMIT   10
        OFFSET  90000
        ) q
JOIN    lookup l
ON      l.id = q.id
ORDER BY
        q.id

Query results

select length(`20110211_late`.`l`.`longtxt`) AS `LENGTH(l.longtxt)` from (select `20110211_late`.`lookup`.`id` AS `id` from `20110211_late`.`lookup` order by `20110211_late`.`lookup`.`id` limit 90000,10) `q` join `20110211_late`.`lookup` `l` where (`20110211_late`.`l`.`id` = `q`.`id`) order by `q`.`id`

No rewrite:

SELECT  LENGTH(longtxt)
FROM    lookup
ORDER BY
        id
LIMIT   10
OFFSET  90000

Query results

select length(`20110211_late`.`lookup`.`longtxt`) AS `LENGTH(longtxt)` from `20110211_late`.`lookup` order by `20110211_late`.`lookup`.`id` limit 90000,10

The query employing the trick runs for same 40 ms, while the straightforward query takes as much as 30 seconds (30,359 ms, to be exact).

Why such a difference?

The reason is that InnoDB, despite the fact it stores the data in the clustered index, is still able to move some data out of the index. This is called external storage.

With COMPACT row format I used to create the tables, InnoDB, when trying to insert a record with two TEXT columns on the page, will try to fit both of them on the page. If this is not possible, then it will split the longest of the records in two parts: the first 768 bytes will be stored on the page and the remaining data will be stored on a separate page (or pages), with a pointer to these data stored in the original clustered index. This will be repeated until all TEXT columns fit on the page of there is no more space there (in which case an error would be thrown).

This means that all TEXT columns shorter than 768 bytes will be stored completely on the page, while those longer can be either split of stored as a whole (with at least first 768 bytes still being on the page).

With the column lengths chosen (1 to 100 and 8K to 8K + 100, accordingly), it can be easily seen that shorttxt will always be stored on-page, while longtxt will always be split (since InnoDB allows at most 8K per record (minus some overhead) to be stored on one page).

Now, this becomes more clear. As with MyISAM, the straightforward query involving longtxt should perform two page lookups per each record scanned: the first one on the clustered index, the second one on the external storage. This takes lots of time and may even spoil the InnoDB cache with unneeded data (which would lead to increased cache miss ratio).

The query on shorttxt is not that bad, since it only requires come extra CPU cycles per record to calculate the string length.

Now, let’s check one more query which orders by a secondary indexed field rather than id:

Secondary index and the short text column

Rewrite:

SELECT  LENGTH(l.shorttxt)
FROM    (
        SELECT  id, value
        FROM    lookup
        ORDER BY
                value
        LIMIT   10
        OFFSET  90000
        ) q
JOIN    lookup l
ON      l.id = q.id
ORDER BY
        q.value

Query results

select length(`20110211_late`.`l`.`shorttxt`) AS `LENGTH(l.shorttxt)` from (select `20110211_late`.`lookup`.`id` AS `id`,`20110211_late`.`lookup`.`value` AS `value` from `20110211_late`.`lookup` order by `20110211_late`.`lookup`.`value` limit 90000,10) `q` join `20110211_late`.`lookup` `l` where (`20110211_late`.`l`.`id` = `q`.`id`) order by `q`.`value`

No rewrite:

SELECT  LENGTH(shorttxt)
FROM    lookup
ORDER BY
        value
LIMIT   10
OFFSET  90000

Query results

select length(`20110211_late`.`lookup`.`shorttxt`) AS `LENGTH(shorttxt)` from `20110211_late`.`lookup` order by `20110211_late`.`lookup`.`value` limit 90000,10

The execution times for these queries vary tenfold: 26 ms vs 266 ms.

As with MyISAM, the secondary index requires an extra lookup to retrieve the actual values from the table (the only difference is that any index is secondary in MyISAM, including that used to police the PRIMARY KEY).

The first query does not perform these row lookups on the skipped records and hence is ten times as fast. It is even faster than queries ordering on the PRIMARY KEY, because the secondary index contains significantly less data than the PRIMARY KEY, holds much more records per page, is hence more shallow and can be traversed faster.

The second query does perform the early row lookups, as usual.

Conclusion

A trick used to avoid early row lookups for the LIMIT … OFFSET queries is useful on InnoDB tables too, though to different extent, depending on the ORDER BY condition and the columns involved:

• It’s very useful on queries involving columns stored off-page (long TEXT, BLOB and VARCHAR columns)
• It’s very useful on ORDER BY conditions served by secondary indexes
• It’s quite useful on moderate sized columns (still stored on page) or CPU-intensive expressions
• It’s almost useless on short columns without complex CPU-intensive processing

Hope that helps.

I’m always glad to answer the questions regarding database queries.

Ask me a question

Well, they have some point. Database engines lack scanner drivers, there is no easy way to do sepia, and magic wand, let’s be honest, is just poor.

However, you can make some New Year paintings with SQL.

Let’s make a New Year clock showing 12 o’clock in Oracle.

#1. Circle

First, we need a circle shape. In Photoshop, you could just select it from the toolbar, but in SQL, we need to use some math.

In math, a circle is defined by this formula: x2 + y2 = R2.

x2 + y2 is, as we know, the square of a distance from the center of the coordinate grid, and R is the radius. This means all points at distance R from the center, which of course is a circle shape.

SQL outputs data in tabular format. We will draw out art string by string, and for each line we need to know where to put the paint (just some ASCII symbols). Since it’s a circle, every line intersects it at most twice, and we shall put the symbols at the places where the circle and the line intersect. To know how many symbols are there from the beginning of the string, this formula is used:

ROUND(SQRT(1 - POWER((level - 21) / 20, 2)) * 20) AS angle

To make our circle more circle-shaped, we need to choose the characters that correspond to the angle of the circle line in each given place. These characters are the most similar to the lines: =/|\

Different characters correspond to different angles. To figure out the angle from the line number, we should use ACOS:

ROUND(ACOS(angle / 20) / 3.1415926 * 4) * SIGN((line - 21) / 20) + 3 AS sign

This would split the circle into several sectors, and for each of these sectors a corresponding character will be chosen.

Finally, we need to avoid the gaps in the lines. To do this, we will not only fill the intersections, but all space from the previous (or next) intersection to the current. To calculated the width of the string on each line that will be filled with the characters, we will use LAG and LEAD, the analytic functions.

Now, let’s put it all together:

WITH    circle AS
        (
        SELECT  1 AS layer,
                line,
                LPAD(RPAD(SUBSTR('=/|\=', sign, 1), width, SUBSTR('=/|\=', sign, 1)), 20 + width - angle, ' ') ||
                RPAD(' ', (angle - width) * 2, ' ') ||
                RPAD(RPAD(SUBSTR('=\|/=', sign, 1), width, SUBSTR('=\|/=', sign, 1)), 20 + width - angle, ' ') AS drawing
        FROM    (
                SELECT  line,
                        angle,
                        ABS
                        (
                        angle -
                        CASE
                        WHEN line < 21 THEN
                                COALESCE(LAG(angle) OVER (ORDER BY line), 0)
                        WHEN line > 21 THEN
                                COALESCE(LEAD(angle) OVER (ORDER BY line), 0)
                        ELSE
                                angle
                        END
                        ) + 1 AS width,
                        ROUND(ACOS(angle / 20) / 3.1415926 * 4)  * SIGN((line - 21) / 20) + 3 AS sign
                FROM    (
                        SELECT  level AS line,
                                ROUND(SQRT(1 - POWER((level - 21) / 20, 2)) * 20) AS angle
                        FROM    dual
                        CONNECT BY
                                level <= 41
                        ) q
                ) q
        )
SELECT  *
FROM    circle

#2. Dial

Now, we shall draw the clock dial on a separate layer (yes, you can draw on layers in SQL).

We will use Roman numerals for the dial. Oracle has an internal function to format the numbers as Roman numerals, but SQL graphics is the first time I’m using this feature in practice.

The principle is the same: we should calculate the angle of each number, then the line and the column that intersect the dial at the given angle and put the number there.

We will only output the lines containing actual numbers. Each line will contain one or two numbers (1 and 6 go on their own lines, the others go in pairs). We can use MIN and MAX to distinguish them.

Here’s what the lines of our dial will look like:

WITH    dial AS
        (
        SELECT  2,
                line,
                RPAD(' ', 20 - angle, ' ') ||
                RPAD(rnf, angle * 2 - LENGTH(rns)) ||
                RPAD(rns, 20 - angle + DECODE(rnf, NULL, 0, LENGTH(rns)) + 1)
        FROM    (
                SELECT  line, angle,
                        DECODE(MAX(h), MIN(h), NULL, TRIM(TO_CHAR(MAX(h), 'RN'))) AS rnf,
                        TRIM(TO_CHAR(MIN(h), 'RN')) AS rns
                FROM    (
                        SELECT  level + 2 AS line,
                                ROUND(SQRT(1 - POWER((level - 19) / 18, 2)) * 18) AS angle
                        FROM    dual
                        CONNECT BY
                                level <= 37
                        ) lines
                JOIN    (
                        SELECT  level AS h, ROUND(-COS(3.141592 * level / 6) * 18) + 21 AS hline
                        FROM    dual
                        CONNECT BY
                                level <= 12
                        ) hours
                ON      hline = line
                GROUP BY
                        line, angle
                ) q
        )
SELECT  *
FROM    dial
ORDER BY
        line

Note that the shape is distorted: this is because of the gaps in the line numbers. It will be taken care of later.

#3. Hands and the center pin

Hands and the center pin are simple: we will use ||| for the hour hand, | for the minute hand and O for the pin. On a New Year clock the hands, fortunately, are vertical and directed in one way.

Here’s the hour hand:

WITH    hourhand AS
        (
        SELECT  3 AS layer,
                level + 10 AS line,
                RPAD(' ', 20, ' ') || '|||' || RPAD(' ', 18, ' ') AS drawing
        FROM    dual
        CONNECT BY
                level <= 10
        )
SELECT  *
FROM    hourhand
ORDER BY
        line

, the minute hand:

WITH    minutehand AS
        (
        SELECT  4 AS layer,
                level + 4 AS line,
                RPAD(' ', 21, ' ') || '|' || RPAD(' ', 19, ' ') AS drawing
        FROM    dual
        CONNECT BY
                level <= 16
        )
SELECT  *
FROM    minutehand
ORDER BY
        line

and the pin:

WITH    pin AS
        (
        SELECT  5 AS layer,
                21 AS line,
                RPAD(LPAD('O', 22, ' '), 41, ' ') AS drawing
        FROM    dual
        )
SELECT  *
FROM    pin
ORDER BY
        line

#4. Merging the layers.

Now we need to merge the layers.

The last layers should go on the foreground, the first ones on the background. Space symbol should be transparent: if there is a character on the background layer, it should be visible through it.

To do it we we apply this trick:

1. Output all rows from all layers in a single query, using UNION ALL
2. Split each row into the individual characters, one character per record

3. Using ROW_NUMBER(), the analytical function, order the characters within each line and columns so that visible characters from higher layers go first, and spaces and characters from the lower layers go last:

   ROW_NUMBER() OVER (PARTITION BY line, col ORDER BY DECODE(cc, ' ', 1, 0), layer DESC) AS rn

4. Filter the characters on rn so that only the topmost character from the each line and column is returned
5. Concatenate all characters line-wise, using a recursive query and SYS_CONNECT_BY_PATH. This function requires a separator, and we will use a question mark, ?, for this purpose, since it’s not used in our art.
6. Remove the separator from the concatenated strings, using REPLACE
7. Output the result

#5. Result

Here’s our query and drawing:

WITH    circle AS
        (
        SELECT  1 AS layer,
                line,
                LPAD(RPAD(SUBSTR('=/|\=', sign, 1), width, SUBSTR('=/|\=', sign, 1)), 20 + width - angle, ' ') ||
                RPAD(' ', (angle - width) * 2, ' ') ||
                RPAD(RPAD(SUBSTR('=\|/=', sign, 1), width, SUBSTR('=\|/=', sign, 1)), 20 + width - angle, ' ') AS drawing
        FROM    (
                SELECT  line,
                        angle,
                        ABS
                        (
                        angle -
                        CASE
                        WHEN line < 21 THEN
                                COALESCE(LAG(angle) OVER (ORDER BY line), 0)
                        WHEN line > 21 THEN
                                COALESCE(LEAD(angle) OVER (ORDER BY line), 0)
                        ELSE
                                angle
                        END
                        ) + 1 AS width,
                        ROUND(ACOS(angle / 20) / 3.1415926 * 4)  * SIGN((line - 21) / 20) + 3 AS sign
                FROM    (
                        SELECT  level AS line,
                                ROUND(SQRT(1 - POWER((level - 21) / 20, 2)) * 20) AS angle
                        FROM    dual
                        CONNECT BY
                                level <= 41
                        ) q
                ) q
        ),
        dial AS
        (
        SELECT  2,
                line,
                RPAD(' ', 20 - angle, ' ') ||
                RPAD(rnf, angle * 2 - LENGTH(rns)) ||
                RPAD(rns, 20 - angle + DECODE(rnf, NULL, 0, LENGTH(rns)) + 1)
        FROM    (
                SELECT  line, angle,
                        DECODE(MAX(h), MIN(h), NULL, TRIM(TO_CHAR(MAX(h), 'RN'))) AS rnf,
                        TRIM(TO_CHAR(MIN(h), 'RN')) AS rns
                FROM    (
                        SELECT  level + 2 AS line,
                                ROUND(SQRT(1 - POWER((level - 19) / 18, 2)) * 18) AS angle
                        FROM    dual
                        CONNECT BY
                                level <= 37
                        ) lines
                JOIN    (
                        SELECT  level AS h, ROUND(-COS(3.141592 * level / 6) * 18) + 21 AS hline
                        FROM    dual
                        CONNECT BY
                                level <= 12
                        ) hours
                ON      hline = line
                GROUP BY
                        line, angle
                ) q
        ),
        hourhand AS
        (
        SELECT  3 AS layer,
                level + 10 AS line,
                RPAD(' ', 20, ' ') || '|||' || RPAD(' ', 18, ' ') AS drawing
        FROM    dual
        CONNECT BY
                level <= 10
        ),
        minutehand AS
        (
        SELECT  4 AS layer,
                level + 4 AS line,
                RPAD(' ', 21, ' ') || '|' || RPAD(' ', 19, ' ') AS drawing
        FROM    dual
        CONNECT BY
                level <= 16
        ),
        pin AS
        (
        SELECT  5 AS layer,
                21 AS line,
                RPAD(LPAD('O', 22, ' '), 41, ' ') AS drawing
        FROM    dual
        ),
        m AS
        (
        SELECT  line, col, cc
        FROM    (
                SELECT  line, col, cc,
                        ROW_NUMBER() OVER (PARTITION BY line, col ORDER BY DECODE(cc, ' ', 1, 0), layer DESC) AS rn
                FROM    (
                        SELECT  cols.*, layers.*, SUBSTR(drawing, col, 1) AS cc
                        FROM    (
                                SELECT  level AS col
                                FROM    dual
                                CONNECT BY
                                        level <= 50
                                ) cols
                        CROSS JOIN
                                (
                                SELECT  *
                                FROM    circle
                                UNION ALL
                                SELECT  *
                                FROM    dial
                                UNION ALL
                                SELECT  *
                                FROM    hourhand
                                UNION ALL
                                SELECT  *
                                FROM    minutehand
                                UNION ALL
                                SELECT  *
                                FROM    pin
                                ) layers
                        )
                )
        WHERE   rn = 1
        )
SELECT  REPLACE(drawing, '?') AS drawing
FROM    (
        SELECT  SYS_CONNECT_BY_PATH(cc, '?') AS drawing, line, CONNECT_BY_ISLEAF AS leaf
        FROM    m mi
        START WITH
                mi.col = 1
        CONNECT BY
                mi.line = PRIOR mi.line
                AND mi.col = PRIOR mi.col + 1
        )
WHERE   leaf = 1
ORDER BY
        line

Jan Suchal asks:

We’ve started playing with PostgreSQL and recursive queries. Looking at example that does basic graph traversal from http://www.postgresql.org/docs/9.0/static/queries-with.html.

We would like to have a view called paths defined like this:

WITH RECURSIVE
        search_graph(id, path) AS
        (
        SELECT  id, ARRAY[id]
        FROM    node
        UNION ALL
        SELECT  g.dest, sg.path || g.dest
        FROM    search_graph sg
        JOIN    graph g
        ON      g.source = sg.id
                AND NOT g.dest = ANY(sg.path)
        )
SELECT  path
FROM    search_graph

By calling

SELECT  *
FROM    paths
WHERE   path[1] = :node_id

we would get all paths from a certain node.

The problem here is with performance. When you want this to be quick you need to add a condition for the anchor part of the UNION like this:

WITH RECURSIVE
        search_graph(id, path) AS
        (
        SELECT  id, ARRAY[id]
        FROM    node
        WHERE   id = :node_id
        UNION ALL
        SELECT  g.dest, sg.path || g.dest
        FROM    search_graph sg
        JOIN    graph g
        ON      g.source = sg.id
                AND NOT g.dest = ANY(sg.path)
        )
SELECT  path
FROM    search_graph

Now it’s perfectly fast, but we cannot create a view because that would only contain paths from one specific node.

Any ideas?

An often overlooked feature of PostgreSQL is its ability to create set-returning functions and use them in SELECT list.

The record will be cross-joined with the set returned by the function and the result of the join will be added to the resultset.

This is best demonstrated with generate_series, probably a most used PostgreSQL set-returning function.

Emulating CROSS APPLY with set-returning functions

Let’s write a simple query that returns numbers from 1 to 3:

SELECT  id
FROM    (
        VALUES
        (1),
        (2),
        (3)
        ) vals (id)

Now, let’s add generate_series(1, id) to the SELECT list. As we can see, each of the three values is passed as an argument to generate_series:

SELECT  id, generate_series(1, id)
FROM    (
        VALUES
        (1),
        (2),
        (3)
        ) vals (id)

We see that each record of the original query was cross-joined with the set returned by the function, and the final resultset now has 6 records instead of 3, since 3 sets, having 1, 2 and 3 records, respectively, were results of these cross-joins.

This is almost what CROSS APPLY does in SQL Server, but with some limitations and caveats.

Limitations of set-returning functions

• One should create a function explicitly. Anonymous blocks won’t work.

• The functions should be written in SQL or C. Procedural language set-returning functions cannot be used in SELECT lists

• If the function return an empty set, the result of the cross join will be an empty set too (i. e. the corresponding record won’t be returned at all). This is exactly how CROSS APPLY behaves in SQL Server.

  However, the latter supports OUTER APPLY which always returns a single record with a NULL in corresponding fields instead of the empty set, and it’s impossible to emulate it in PostgreSQL using this syntax.

  A workaround would be to develop the function so that it would return a single NULL instead of an empty set.

• The function should be marked VOLATILE so that the optimizer would execute it each time it’s called. This is especially important when the function is used multiple times in the same SELECT list, like this:

  SELECT  generate_series(1, 3), generate_series(1, 2)
  

  generate_series	generate_series
  1	1
  2	2
  3	1
  1	2
  2	1
  3	2

  This query correctly returns 6 records.

  SELECT  generate_series(1, 3), generate_series(1, 3)
  

  generate_series	generate_series
  1	1
  2	2
  3	3

  This query incorrectly returns 3 records (it should return 9). This is because function is not reevaluated.

Parametrizing the path query

Now, we are informed enough to create our own function and test it. To do this, we will first create some sample tables:

Table creation details

CREATE TABLE node
        (
        id BIGINT NOT NULL PRIMARY KEY,
        name VARCHAR(100) NOT NULL
        );

CREATE TABLE graph
        (
        source BIGINT NOT NULL,
        dest BIGINT NOT NULL,
        data FLOAT NOT NULL,
        PRIMARY KEY (source, dest)
        );

CREATE INDEX
        ix_graph_dest
ON      graph (dest);

SELECT  SETSEED(0.20101224);

INSERT
INTO    node
SELECT  s, 'Node ' || s
FROM    generate_series(1, 10000) s;

INSERT
INTO    graph (source, dest, data)
SELECT  source, dest, RANDOM()
FROM    (
        SELECT  DISTINCT
                CEIL(RANDOM() * 10000) AS source,
                CEIL(RANDOM() * 10000) AS dest
        FROM    generate_series(1, 10000)
        ) q
WHERE   source <> dest

There are 10,000 nodes with 9,996 random paths between them.

Here’s what the function would look like:

CREATE OR REPLACE FUNCTION fn_search_graph_cte(parent BIGINT)
RETURNS TABLE
        (
        path BIGINT[]
        )
AS
$$
        WITH RECURSIVE
                search_graph(id, path) AS
                (
                SELECT  $1, ARRAY[$1]
                UNION ALL
                SELECT  g.dest, sg.path || g.dest
                FROM    search_graph sg
                JOIN    graph g
                ON      g.source = sg.id
                        AND NOT g.dest = ANY(sg.path)
                )
        SELECT  path
        FROM    search_graph;
$$
LANGUAGE 'sql'
VOLATILE;

The anchor part of the recursive CTE is parametrized.

Paths from a single node

Now, let’s compare performance of the queries that select paths from a single node:

WITH RECURSIVE
        search_graph(id, path) AS
        (
        SELECT  id, ARRAY[id]
        FROM    node
        UNION ALL
        SELECT  g.dest, sg.path || g.dest
        FROM    search_graph sg
        JOIN    graph g
        ON      g.source = sg.id
                AND NOT g.dest = ANY(sg.path)
        )
SELECT  path::TEXT
FROM    search_graph
WHERE   path[1] = 69

CTE Scan on search_graph  (cost=151096.75..187130.22 rows=7999 width=32)
  Filter: (path[1] = 69)
  CTE search_graph
    ->  Recursive Union  (cost=0.00..151096.75 rows=1599710 width=40)
          ->  Seq Scan on node  (cost=0.00..164.00 rows=10000 width=8)
          ->  Hash Join  (cost=288.91..11893.85 rows=158971 width=40)
                Hash Cond: (sg.id = g.source)
                Join Filter: (g.dest <> ALL (sg.path))
                ->  WorkTable Scan on search_graph sg  (cost=0.00..2000.00 rows=100000 width=40)
                ->  Hash  (cost=163.96..163.96 rows=9996 width=16)
                      ->  Seq Scan on graph g  (cost=0.00..163.96 rows=9996 width=16)

This query materializes the whole CTE and then filters it for the paths beginning from 69. It takes more than 500 ms.

SELECT  fn_search_graph_cte(id)::TEXT
FROM    node
WHERE   id = 69

Index Scan using node_pkey on node  (cost=0.00..8.52 rows=1 width=8)
  Index Cond: (id = 69)

This uses the function. Since the anchor part of the CTE has only one record, this is much faster and completes in 3 ms.

All paths

Let’s check how long does it take to return all paths.

First, let’s use the CTE:

WITH RECURSIVE
        search_graph(id, path) AS
        (
        SELECT  id, ARRAY[id]
        FROM    node
        UNION ALL
        SELECT  g.dest, sg.path || g.dest
        FROM    search_graph sg
        JOIN    graph g
        ON      g.source = sg.id
                AND NOT g.dest = ANY(sg.path)
        )
SELECT  COUNT(*), AVG(ARRAY_LENGTH(path, 1))
FROM    search_graph

Aggregate  (cost=191089.51..191089.52 rows=1 width=32)
  CTE search_graph
    ->  Recursive Union  (cost=0.00..151096.75 rows=1599710 width=40)
          ->  Seq Scan on node  (cost=0.00..164.00 rows=10000 width=8)
          ->  Hash Join  (cost=288.91..11893.85 rows=158971 width=40)
                Hash Cond: (sg.id = g.source)
                Join Filter: (g.dest <> ALL (sg.path))
                ->  WorkTable Scan on search_graph sg  (cost=0.00..2000.00 rows=100000 width=40)
                ->  Hash  (cost=163.96..163.96 rows=9996 width=16)
                      ->  Seq Scan on graph g  (cost=0.00..163.96 rows=9996 width=16)
  ->  CTE Scan on search_graph  (cost=0.00..31994.20 rows=1599710 width=32)

Now, the function:

SELECT  COUNT(*), AVG(ARRAY_LENGTH(path, 1))
FROM    (
        SELECT  fn_search_graph_cte(id) AS path
        FROM    node
        ) q

Aggregate  (cost=2814.01..2814.02 rows=1 width=32)
  ->  Seq Scan on node  (cost=0.00..2664.00 rows=10000 width=8)

We see that the function introduces some overhead: it needs to be called 10,000 times and this takes 2 seconds (as opposed to 500 ms for the plain CTE). This means that the function is more efficient if the total number of the root nodes is less than 25% of all nodes.

Conclusion

A set-returning function is a good replacement for a view over a recursive CTE whose anchor part cannot be easily parametrized.

The sets returned by the function, when called in a SELECT list, are cross-joined with the corresponding records. When the filter on the CTE is selective, the function will only be applied few times to the records satisfying the filter.

Call to a function, however, introduces some overhead which should be taken into account. In some cases, when a filter is not very selective, it is better to use the benefits of set-based operations to generate the view and then filter it than to call a function multiple times, adding the overhead of each call.

Hope that helps.

I’m always glad to answer the questions regarding database queries.

Ask me a question

#10. Searching for a NULL

SELECT  *
FROM    a
WHERE   a.column = NULL

In SQL, a NULL is never equal to anything, even another NULL. This query won’t return anything and in fact will be thrown out by the optimizer when building the plan.

When searching for NULL values, use this instead:

SELECT  *
FROM    a
WHERE   a.column IS NULL

#9. LEFT JOIN with additional conditions

SELECT  *
FROM    a
LEFT JOIN
        b
ON      b.a = a.id
WHERE   b.column = 'something'

A LEFT JOIN is like INNER JOIN except that it will return each record from a at least once, substituting missing fields from b with NULL values, if there are no actual matching records.

The WHERE condition, however, is evaluated after the LEFT JOIN so the query above checks column after it had been joined. And as we learned earlier, no NULL value can satisfy an equality condition, so the records from a without corresponding record from b will unavoidably be filtered out.

Essentially, this query is an INNER JOIN, only less efficient.

To match only the records with b.column = 'something' (while still returning all records from a), this condition should be moved into ON clause:

SELECT  *
FROM    a
LEFT JOIN
        b
ON      b.a = a.id
        AND b.column = 'something'

#8. Less than a value but not a NULL

Quite often I see the queries like this:

SELECT  *
FROM    b
WHERE   b.column < 'something'
        AND b.column IS NOT NULL

This is actually not an error: this query is valid and will do what’s intended. However, IS NOT NULL here is redundant.

If b.column is a NULL, then b.column < 'something' will never be satisfied, since any comparison to NULL evaluates to a boolean NULL and does not pass the filter.

It is interesting that this additional NULL check is never used for greater than queries (like in b.column > 'something').

This is because NULL go first in ORDER BY in MySQL and hence are incorrectly considered less than any other value by some people.

This query can be simplified:

SELECT  *
FROM    b
WHERE   b.column < 'something'

and will still never return a NULL in b.column.

#7. Joining on NULL

SELECT  *
FROM    a
JOIN    b
ON      a.column = b.column

When column is nullable in both tables, this query won't return a match of two NULLs for the reasons described above: no NULLs are equal.

Here's a query to do that:

SELECT  *
FROM    a
JOIN    b
ON      a.column = b.column
        OR (a.column IS NULL AND b.column IS NULL)

MySQL's optimizer treats this as an equijoin and provides a special join condition, ref_or_null.

#6. NOT IN with NULL values

SELECT  a.*
FROM    a
WHERE   a.column NOT IN
        (
        SELECT column
        FROM    b
        )

This query will never return anything if there is but a single NULL in b.column. As with other predicates, both IN and NOT IN against NULL evaluate to NULL.

This should be rewritten using a NOT EXISTS:

SELECT  a.*
FROM    a
WHERE   NOT EXISTS
        (
        SELECT NULL
        FROM    b
        WHERE   b.column = a.column
        )

Unlike IN, EXISTS always evaluates to either true or false.

#5. Ordering random samples

SELECT  *
FROM    a
ORDER BY
        RAND(), column
LIMIT 10

This query attempts to select 10 random records ordered by column.

ORDER BY orders the output lexicographically: that is, the records are only ordered on the second expression when the values of the first expression are equal.

However, the results of RAND() are, well, random. It's infeasible that the values of RAND() will match, so ordering on column after RAND() is quite useless.

To order the randomly sampled records, use this query:

SELECT  *
FROM    (
        SELECT  *
        FROM    mytable
        ORDER BY
                RAND()
        LIMIT 10
        ) q
ORDER BY
        column

#4. Sampling arbitrary record from a group

This query intends to select one column from each group (defined by grouper)

SELECT  DISTINCT(grouper), a.*
FROM    a

DISTINCT is not a function, it's a part of SELECT clause. It applies to all columns in the SELECT list, and the parentheses here may just be omitted. This query may and will select the duplicates on grouper (if the values in at least one of the other columns differ).

Sometimes, it's worked around using this query (which relies on MySQL's extensions to GROUP BY):

SELECT  a.*
FROM    a
GROUP BY
        grouper

Unaggregated columns returned within each group are arbitrarily taken.

At first, this appears to be a nice solution, but it has quite a serious drawback. It relies on the assumption that all values returned, though taken arbitrarily from the group, will still belong to one record.

Though with current implementation is seems to be so, it's not documented and can be changed in any moment (especially if MySQL will ever learn to apply index_union after GROUP BY). So it's not safe to rely on this behavior.

This query would be easy to rewrite in a cleaner way if MySQL supported analytic functions. However, it's still possible to make do without them, if the table has a PRIMARY KEY defined:

SELECT  a.*
FROM    (
        SELECT  DISTINCT grouper
        FROM    a
        ) ao
JOIN    a
ON      a.id =
        (
        SELECT  id
        FROM    a ai
        WHERE   ai.grouper = ao.grouper
        LIMIT 1
        )

#3. Sampling first record from a group

This is a variation of the previous query:

SELECT  a.*
FROM    a
GROUP BY
        grouper
ORDER BY
        MIN(id) DESC

Unlike the previous query, this one attempts to select the record holding the minimal id.

Again: it is not guaranteed that the unaggregated values returned by a.* will belong to a record holding MIN(id) (or even to a single record at all).

Here's how to do it in a clean way:

SELECT  a.*
FROM    (
        SELECT  DISTINCT grouper
        FROM    a
        ) ao
JOIN    a
ON      a.id =
        (
        SELECT  id
        FROM    a ai
        WHERE   ai.grouper = ao.grouper
        ORDER BY
                ai.grouper, ai.id
        LIMIT 1
        )

This query is just like the previous one but with ORDER BY added to ensure that the first record in id order will be returned.

#2. IN and comma-separated list of values

This query attempts to match the value of column against any of those provided in a comma-separated string:

SELECT  *
FROM    a
WHERE   column IN ('1, 2, 3')

This does not work because the string is not expanded in the IN list.

Instead, if column column is a VARCHAR, it is compared (as a string) to the whole list (also as a string), and of course will never match. If column is of a numeric type, the list is cast into the numeric type as well (and only the first item will match, at best).

The correct way to deal with this query would be rewriting it as a proper IN list

SELECT  *
FROM    a
WHERE   column IN (1, 2, 3)

, or as an inline view:

SELECT  *
FROM    (
        SELECT  1 AS id
        UNION ALL
        SELECT  2 AS id
        UNION ALL
        SELECT  3 AS id
        ) q
JOIN    a
ON      a.column = q.id

, but this is not always possible.

To work around this without changing the query parameters, one can use FIND_IN_SET:

SELECT  *
FROM    a
WHERE   FIND_IN_SET(column, '1,2,3')

This function, however, is not sargable and a full table scan will be performed on a.

#1. LEFT JOIN with COUNT(*)

SELECT  a.id, COUNT(*)
FROM    a
LEFT JOIN
        b
ON      b.a = a.id
GROUP BY
        a.id

This query intends to count number of matches in b for each record in a.

The problem is that COUNT(*) will never return a 0 in such a query. If there is no match for a certain record in a, the record will be still returned and counted.

COUNT should be made to count only the actual records in b. Since COUNT(*), when called with an argument, ignores NULLs, we can pass b.a to it. As a join key, it can never be a null in an actual match, but will be if there were no match:

SELECT  a.id, COUNT(b.a)
FROM    a
LEFT JOIN
        b
ON      b.a = a.id
GROUP BY
        a.id

P.S. In case you were wondering: no, the pictures don't have any special meaning. I just liked them.