EXPLAIN EXTENDED » Miscellaneous http://explainextended.com How to create fast database queries Mon, 02 Jan 2012 00:31:26 +0000 en hourly 1 http://wordpress.org/?v=3.3.2 Happy New Year! http://explainextended.com/2011/12/31/happy-new-year-3/ http://explainextended.com/2011/12/31/happy-new-year-3/#comments Sat, 31 Dec 2011 19:00:56 +0000 Quassnoi http://explainextended.com/?p=5408 This winter is anomalously warm in Europe, there is no snow and no New Year mood. So today we will be drawing a snowflake in PostgreSQL.

#1. A little theory

Core of a snowflake is six large symmetrical ice crystals growing from the common center. Out of these larger crystals other, smaller, crystals grow.

The overall shape of the snowflake is defined by how large do crystals grow and where exactly are they attached to each other.

These things are defined by fluctuations in air and temperature conditions around the snowflake. Because the flake itself is very small, in any given moment the conditions are nearly identical around each crystal, that’s why the offspring crystals pop up in almost same places and grow to almost same lengths. Different flakes, though, constantly move to and from each other and are subject to very different fluctuations, and that’s why they grow so unique.

Except for the root crystals (of which there are six), the child icicles grow in symmetrical pairs. More than that, each branch grows their own children (also in pairs), so on each step there are twice as many crystals, but they all share almost same length and angle. This gives the snowflake its symmetrical look.

So we can easily see that, despite the fact there may be many child crystals, the shape of a snowflake is defined by a relatively small number of parameters: how many children each crystal produces, where are they attached to it, at which angle they grow and to which length.

Now, let’s try to model it.

#2. Defining parameters

To begin with, we will assume the length of the initial larger crystal as 1, and its angle as 0. Later we may easily scale and rotate the crystal.

There can be any number of child crystals (or, rather, pairs of child crystals), each pair attaching to its parent in a random place and growing at random angle. However, this places and angles, though random, are shared across the snowflake: if one pair grows here and at this angle, the similar pairs will grow on the same place and at the same angle on their parents.

This means that there are 6 * (2 ^ level) twins of each crystal, where level is the number of its ancestors. There are 6 root crystals, 12 first-level crystals in the first first-level pairs (each growing in the same place at the same angle to the same length on root crystals), 12 first-level crystals in the second first-level pairs (again, sharing all parameters with themselves but not with the first pairs), 24 first-first grandchildren, 24 first-second grandchildren etc.

So, to define the snowflake shape, we need to define the number of pairs on each step, and for each pair define its place on the parent crystal, angle and length.

These parameters are random but have some constraints. Child crystals are shorter than their parents; they grow at sharp angles, and the number of pairs is limited. Since the parameters of each crystal depend on its parent, we would need a recursive query for that:

SELECT  SETSEED(0.20111231);

WITH    RECURSIVE
        params (id, cut, len, alpha, level, spikes) AS
        (
        SELECT  ARRAY[0::INT], 0::DOUBLE PRECISION, 1::DOUBLE PRECISION, 0::DOUBLE PRECISION, 1, FLOOR(RANDOM() * 4)::INT AS spikes
        UNION ALL
        SELECT  id, cut, len, alpha, level, FLOOR(RANDOM() * 4)::INT AS spikes
        FROM    (
                SELECT  id || generate_series(0, spikes) AS id,
                        RANDOM() AS cut, len * RANDOM() * 0.5 AS len, (PI() / 2) * (0.3 + RANDOM() * 0.3) AS alpha, level + 1 AS level
                FROM    params
                ) q
        WHERE   level <= 3
        )
SELECT  id::TEXT, cut, len, alpha, level, spikes
FROM    params;
id cut len alpha level spikes
{0} 0 1 0 1 2
{0,0} 0.356672627385706 0.285511903231964 0,895624824686616 2 2
{0,1} 0.561545127537102 0.170363144949079 0.618425840960865 2 3
{0,2} 0.475528724957258 0.394589512841776 0.81481895943341 2 0
{0,0,0} 0.667989769019186 0.0429738523301687 0.818223701208857 3 1
{0,0,1} 0.325173982884735 0.00710072719083308 0.517746497950111 3 0
{0,0,2} 0.181089712772518 0.0441518332901016 0.789388660278232 3 0
{0,1,0} 0.272290788125247 0.0784957594457395 0.522800933603073 3 0
{0,1,1} 0.559544350020587 0.0571536974970293 0.713263419194096 3 3
{0,1,2} 0.642731287050992 0.0842548574598808 0.782757860467376 3 1
{0,1,3} 0.0450216801837087 0.0163872951631622 0.590039109739568 3 0
{0,2,0} 0.289853980299085 0.173285180190047 0.896723437835692 3 1
12 rows fetched in 0.0005s (0.0047s)

Here, id is an array defining the path to the crystal pairs: {0} is the root branch, {0, 0} is the first child pair, {0, 1, 0} is the first child of the second child etc.

cut defines the position where the crystals are attached to the parent: 0.356672627385706 in the second row means the first child will grow at 35.6% of the root branches’ length.

len is the child length (in respect to the root branch, not immediate parent), angle is the angle at which they grow (in respect to the immediate parent), and spikes is the number of children pairs for the given crystal. level, I hope, is self-explanatory.

#3. Defining crystal coordinates

Now that we have the shape of our flake defined we need to build the actual coordinates for each crystal. To do this we again would need a recursive query.

First, we should make a set of branches on each level. We would start from a single root branch (it will be easy to clone it later) and generate a set of branches on each level. Each record in this set would correspond to an actual snowflake crystal:

WITH    RECURSIVE
        params (id, cut, len, alpha, level, spikes) AS
        (
        SELECT  ARRAY[0::INT], 0::DOUBLE PRECISION, 1::DOUBLE PRECISION, 0::DOUBLE PRECISION, 1, FLOOR(RANDOM() * 4)::INT AS spikes
        UNION ALL
        SELECT  id, cut, len, alpha, level, FLOOR(RANDOM() * 4)::INT AS spikes
        FROM    (
                SELECT  id || generate_series(0, spikes) AS id,
                        RANDOM() AS cut, len * RANDOM() * 0.5 AS len, (PI() / 2) * (0.3 + RANDOM() * 0.3) AS alpha, level + 1 AS level
                FROM    params
                ) q
        WHERE   level <= 3
        ),
        tree AS
        (
        SELECT  *, generate_series(0, (1 << (level - 1)) - 1) AS branch
        FROM    params
        )
SELECT  id::TEXT, cut, len, alpha, level, spikes, branch
FROM    tree;

View query output

This is just a copy of the parameters recordset with each parameter duplicated as many times as the crystal level requires. There is one root branch, two copies of level 1 branches, four copies of level 2 branches etc. Each instance is defined by the parameter branch.

To calculate coordinates of each crystal we need to traverse to it from the top.

The coordinates of the root branch are known: they are (0, 0), (1, 0) (by definition).

To build the first pair (two crystals, each 28.5% long, growing at 35.6% from the beginning of the root branch at angles 51° 18′ and −51° 18′, respectively) we would need to take the coordinates of the parent, find the start point (it would be (Xs + (Xe - Xs) * cut, Ys + (Ye - Ys) * cut) and then the coordinates of the end point (by adding len * COS(α) and len * SIN(α) to X and Y of the start point, respectively). α in this formula is in respect to the coordinate grid, not to the parent, and to find it we just need to sum all angles from the ancestors.

This is best done with another recursive query. On each step we should find immediate children of the current parent.

This can be easily achieved using the following join condition: child.id > parent.id AND child.id <= parend.id || spikes AND ARRAY_LENGTH(child.level, 1) = ARRAY_LENGTH(parent.level, 1) + 1. This condition employs PostgreSQL array arithmetics: if we have parent.id = {0, 2, 1} with 3 children (spikes = 2), then the condition would return {0, 2, 1, 0}, {0, 2, 1, 1} and {0, 2, 1, 2}. This hierarchy model is called materialized path, and some day I will write a post about it.

Since the id alone defines the parameters, not the instance, we need to add once more condition to find the instances. Crystals 0 and 1 would have child pairs 0, 1 and 2, 3, respectively, so we'll include child.branch BETWEEN p.branch * 2 AND child.branch * 2 + 1 into the join condition.

One more thing to do is to find whether we should add positive or negative angle. It's simple: even branches are negative, odd ones are positive.

And here's the query:

SELECT  SETSEED(0.20111231);

WITH    RECURSIVE
        params (id, cut, len, alpha, level, spikes) AS
        (
        SELECT  ARRAY[0::INT], 0::DOUBLE PRECISION, 1::DOUBLE PRECISION, 0::DOUBLE PRECISION, 1, FLOOR(RANDOM() * 4)::INT AS spikes
        UNION ALL
        SELECT  id, cut, len, alpha, level, FLOOR(RANDOM() * 4)::INT AS spikes
        FROM    (
                SELECT  id || generate_series(0, spikes) AS id,
                        RANDOM() AS cut, len * RANDOM() * 0.5 AS len, (PI() / 2) * (0.3 + RANDOM() * 0.3) AS alpha, level + 1 AS level
                FROM    params
                ) q
        WHERE   level <= 3
        ),
        tree AS
        (
        SELECT  *, generate_series(0, (1 << (level - 1)) - 1) AS branch
        FROM    params
        ),
        points AS
        (
        SELECT  id,
                0::double precision AS xs,
                0::double precision AS ys,
                len AS xe,
                0::double precision AS ye,
                0::double precision AS alpha,
                branch,
                level,
                spikes
        FROM    tree
        WHERE   id = ARRAY[0]
        UNION ALL
        SELECT  id,
                xs + (xe - xs) * cut AS xs,
                ys + (ye - ys) * cut AS ys,
                xs + (xe - xs) * cut + len * COS(alpha) AS xe,
                ys + (ye - ys) * cut + len * SIN(alpha) AS ye,
                alpha,
                branch,
                level,
                spikes
        FROM    (
                SELECT  t.id, p.id || t.id AS uid, xs, xe, ys, ye, cut, len,
                        p.alpha + t.alpha * ((t.branch % 2) * 2 - 1) AS alpha,
                        t.branch, t.level, t.spikes
                FROM    points p
                JOIN    tree t
                ON      t.id > p.id
                        AND t.id <= (p.id || p.spikes)
                        AND ARRAY_LENGTH(t.id, 1) = p.level + 1
                        AND t.branch BETWEEN p.branch * 2 AND p.branch * 2 + 1
                ) q
        )
SELECT  id::TEXT, xs, ys, xe, ye, alpha, branch, level
FROM    points

View query output

Now, we have coordinates for each crystal.

#4. Visualizing

The most tedious part about SQL graphics is visualizing them. To do this, we'll employ PostgreSQL's geometrical functions.

Each crystal can be represented as a path between its start and end points. This can be used by constructing a line segment (LSEG(POINT, POINT)) using two point constructors (POINT(DOUBLE PRECISION, DOUBLE PRECISION)) and converting it to a path. Unfortunately, PostgreSQL does not allow direct lseg to path conversion but the latter can be easily constructed from the TEXT representation of an lseg.

We have six root branches so each crystal should be cloned to make six copies. It's easiest to make it by using PATH * POINT operator: it rotates and scales the PATH around (0, 0) so that (1, 0) becomes POINT. To construct the points, we will generate six rotation angles with step of 60° and will multiply the path by POINT(COS(α), SIN(α)). These multiplications will preserve lengths.

Finally we need to actually display the snowflake on the screen. To do this, we will generate a set of 80 × 80 records(x and y), defining the grid from (-1, -1) to (1, 1) with step of 1/40 units. Then we'll see if there is at least one crystal within distance of 1/40 units from each cell on the grid (using POINT <-> PATH distance operator and EXISTS). If there is, we will return a number sign (#) for this cell, otherwise a space.

Then we'll group the cells by lines (y), concatenate columns (using ARRAY_AGG and ARRAY_TO_STRING) and output the lines.

#5. The snowflake

And here's our snowflake:

SELECT  SETSEED(0.20111231);

WITH    RECURSIVE
        params (id, cut, len, alpha, level, spikes) AS
        (
        SELECT  ARRAY[0::INT], 0::DOUBLE PRECISION, 1::DOUBLE PRECISION, 0::DOUBLE PRECISION, 1, FLOOR(RANDOM() * 4)::INT AS spikes
        UNION ALL
        SELECT  id, cut, len, alpha, level, FLOOR(RANDOM() * 4)::INT AS spikes
        FROM    (
                SELECT  id || generate_series(0, spikes) AS id,
                        RANDOM() AS cut, len * RANDOM() * 0.5 AS len, (PI() / 2) * (0.3 + RANDOM() * 0.3) AS alpha, level + 1 AS level
                FROM    params
                ) q
        WHERE   level <= 3
        ),
        tree AS
        (
        SELECT  *, generate_series(0, (1 << (level - 1)) - 1) AS branch
        FROM    params
        ),
        points AS
        (
        SELECT  id,
                0::double precision AS xs,
                0::double precision AS ys,
                len AS xe,
                0::double precision AS ye,
                0::double precision AS alpha,
                branch,
                level,
                spikes
        FROM    tree
        WHERE   id = ARRAY[0]
        UNION ALL
        SELECT  id,
                xs + (xe - xs) * cut AS xs,
                ys + (ye - ys) * cut AS ys,
                xs + (xe - xs) * cut + len * COS(alpha) AS xe,
                ys + (ye - ys) * cut + len * SIN(alpha) AS ye,
                alpha,
                branch,
                level,
                spikes
        FROM    (
                SELECT  t.id, p.id || t.id AS uid, xs, xe, ys, ye, cut, len,
                        p.alpha + t.alpha * ((t.branch % 2) * 2 - 1) AS alpha,
                        t.branch, t.level, t.spikes
                FROM    points p
                JOIN    tree t
                ON      t.id > p.id
                        AND t.id <= (p.id || p.spikes)
                        AND ARRAY_LENGTH(t.id, 1) = p.level + 1
                        AND t.branch BETWEEN p.branch * 2 AND p.branch * 2 + 1
                ) q
        ),
        lines AS
        (
        SELECT  PATH(LSEG(POINT(xs, ys), POINT(xe, ye))::TEXT) * POINT(COS(RADIANS(turn)), SIN(RADIANS(turn))) AS line,
                *
        FROM    points
        CROSS JOIN
                generate_series(0, 300, 60) AS turn
        )
SELECT  ARRAY_TO_STRING
                (
                ARRAY_AGG
                        (
                        CASE
                                EXISTS
                                (
                                SELECT  NULL
                                FROM    lines
                                WHERE   POINT(x::DOUBLE PRECISION / scale, y::DOUBLE PRECISION / scale) <-> line <= (0.7 / scale)
                                )
                        WHEN    TRUE THEN
                                '#'
                        ELSE    ' '
                        END
                        ORDER BY
                                x
                        ),
                ''
                )
FROM    (
        SELECT  *,
                generate_series(-scale, scale - 1) AS y
        FROM    (
                SELECT  scale, generate_series(-scale, scale - 1) AS x
                FROM    (
                        VALUES
                        (40)
                        ) q (scale)
                ) q
        ) q
GROUP BY
        y
ORDER BY
        y
array_to_string
# #
## ##
# #
## # # ##
# # # #
## ## ## ##
## # # ##
# # # #
## ## ## ## ## ##
# ######## ######## #
### #### # # #### ###
## ###### # # # # ###### ##
########### ### ### ###########
################ ################
####### ###### ###### #######
#### ######### ## ### ### ## ######### ####
###### ## ## # ## ## # ## ## ######
####### ### # # ### #######
####### ## ## #######
### ## ### ### ## ###
#### ##### ##### ####
###### # ### ### # ######
## ########## ########## ##
# # ##### ##### # #
## ## ## ##
## # ## ## # ##
## ## ## ## ## ##
## #### ## ## #### ##
## #### # # #### ##
#### #### ## ## #### ####
## ### ## # # ## ### ##
#### ## ## ## ## ## ## ####
######### ### ## ## ### #########
####### ## ### # # ### ## #######
##### # ## ### ## # #####
################################################################################
##### # ## ### ## # #####
####### ## ### # # ### ## #######
######### ### ## ## ### #########
#### ## ## ## ## ## ## ####
## ### ## # # ## ### ##
#### #### ## ## #### ####
## #### # # #### ##
## #### ## ## #### ##
## ## ## ## ## ##
## # ## ## # ##
## ## ## ##
# # ##### ##### # #
## ########## ########## ##
###### # ### ### # ######
#### ##### ##### ####
### ## ### ### ## ###
####### ## ## #######
####### ### # # ### #######
###### ## ## # ## ## # ## ## ######
#### ######### ## ### ### ## ######### ####
####### ###### ###### #######
################ ################
########### ### ### ###########
## ###### # # # # ###### ##
### #### # # #### ###
# ######## ######## #
## ## ## ## ## ##
# # # #
## # # ##
## ## ## ##
# # # #
## # # ##
# #
## ##
# #
80 rows fetched in 0.0010s (2.7344s)

#6. Some more unique snowflakes

View query

array_to_string
#### ####
###### ######
#### ####
## ##
# #
# # # #
## ## ## ##
# ## ## #
# ## # # ## #
## # ## ## # ##
## ## ## ## ## ## ## ##
###### ## # # ## ######
######## # # # # ########
## ###### ### ### ###### ##
# ###### ### ###### #
## ### ##
# # #
## ### ##
## # ### # ##
# ## ##### ## #
## ### ### ### ##
# ##### ##### #
## #### #### ##
# ##### ##### #
## ########### ##
#### ## ########### ## ####
# #### # #### #### # #### #
# ### #### ## #### #### ## #### ### #
# ####### #### # ## # # ## # #### ####### #
# # ######## ## ## # ### # ## ## ######## #
### ## ####### ### ### ### ####### ## ##
### ### ############# # ############# ### ##
## ## ###### ############### ###### ## ##
## #### ##### ## ## # ## ## # ## ## ##### #### ##
## ######### # ############# # ######### ##
## #### ######### ########### ######### #### ##
## ####### ######## ######## ####### ##
## #### ####### ####### #### ##
## ######## # # ######## ##
## ## ### ### ### ## ##
################################################################################
## ## ### ### ### ## ##
## ######## # # ######## ##
## #### ####### ####### #### ##
## ####### ######## ######## ####### ##
## #### ######### ########### ######### #### ##
## ######### # ############# # ######### ##
## #### ##### ## ## # ## ## # ## ## ##### #### ##
## ## ###### ############### ###### ## ##
### ### ############# # ############# ### ##
### ## ####### ### ### ### ####### ## ##
# # ######## ## ## # ### # ## ## ######## #
# ####### #### # ## # # ## # #### ####### #
# ### #### ## #### #### ## #### ### #
# #### # #### #### # #### #
#### ## ########### ## ####
## ########### ##
# ##### ##### #
## #### #### ##
# ##### ##### #
## ### ### ### ##
# ## ##### ## #
## # ### # ##
## ### ##
# # #
## ### ##
# ###### ### ###### #
## ###### ### ### ###### ##
######## # # # # ########
###### ## # # ## ######
## ## ## ## ## ## ## ##
## # ## ## # ##
# ## # # ## #
# ## ## #
## ## ## ##
# # # #
# #
## ##
#### ####
###### ######
80 rows fetched in 0.0010s (2.0781s)

View query

array_to_string
## ## ## ## ## ##
##### ## ## #####
#### # # ####
######## # # ########
##### ## ## # # ## ## #####
# ### # # # # ### #
## ### ### ##
##### #####
### ###
# #
## ##
## ##
# #
## # # # # ##
# # ### # # # ### # #
## ###### # ### # ###### ##
# ### # #### #### # ### #
## ## #### #### ## ##
## ## ### ## ## ### ## ##
### # # ## ## # # ###
##### ## # ## ## # ## #####
###### # ## # # ## # ######
# ### #### # # #### ### #
## # ####### # # ####### # ##
##### ##### #### #### ##### #####
#### ######## ######## ####
####### ## # ## ## # ## #######
### ##### #### #### ##### ###
# ### ### ### ### #
# ######## ######## #
## # ## ### ### ## # ##
#### ##### ##### ####
### ##### ##### ###
######## ##### ##### ########
### # ## ## #### #### ## ## # ###
### ## # # # # ## ###
##### #### ## ## #### #####
###### ## ## ## ## ######
## ######### ## # # ## ######### #
### ### ### #### ### #### ### ### ###
################################################################################
### ### ### #### ### #### ### ### ###
## ######### ## # # ## ######### #
###### ## ## ## ## ######
##### #### ## ## #### #####
### ## # # # # ## ###
### # ## ## #### #### ## ## # ###
######## ##### ##### ########
### ##### ##### ###
#### ##### ##### ####
## # ## ### ### ## # ##
# ######## ######## #
# ### ### ### ### #
### ##### #### #### ##### ###
####### ## # ## ## # ## #######
#### ######## ######## ####
##### ##### #### #### ##### #####
## # ####### # # ####### # ##
# ### #### # # #### ### #
###### # ## # # ## # ######
##### ## # ## ## # ## #####
### # # ## ## # # ###
## ## ### ## ## ### ## ##
## ## #### #### ## ##
# ### # #### #### # ### #
## ###### # ### # ###### ##
# # ### # # # ### # #
## # # # # ##
# #
## ##
## ##
# #
### ###
##### #####
## ### ### ##
# ### # # # # ### #
##### ## ## # # ## ## #####
######## # # ########
#### # # ####
##### ## ## #####
80 rows fetched in 0.0009s (4.4531s)

View query

array_to_string
# #
## ##
# #
## ##
# #
## # # ##
### ###
###### ######
#### ####
## ##
## ##
# #
## ##
## ##
# #
## ##
# #
## ##
# #
## ##
## ##
# #
## ##
# #
## ##
## ##
## ##
## ##
# #
## ##
# #
## ##
## ##
### # # ###
### ### ###
################################################################################
### ### ###
### # # ###
## ##
## ##
# #
## ##
# #
## ##
## ##
## ##
## ##
# #
## ##
# #
## ##
## ##
# #
## ##
# #
## ##
# #
## ##
## ##
# #
## ##
## ##
#### ####
###### ######
### ###
## # # ##
# #
## ##
# #
## ##
# #
80 rows fetched in 0.0009s (1.2188s)
Happy New Year!

Previous New Year posts:

]]> http://explainextended.com/2011/12/31/happy-new-year-3/feed/ 6 Shared Plan and Algorithm Network Cache (SPANC) http://explainextended.com/2011/04/01/shared-plan-and-algorithm-network-cache-spanc/ http://explainextended.com/2011/04/01/shared-plan-and-algorithm-network-cache-spanc/#comments Fri, 01 Apr 2011 19:00:49 +0000 Quassnoi http://explainextended.com/?p=5314 Due to the nature of my work I have to deal with various database systems.

While SQL is more or less standardized, the optimizers are implemented differently in the different systems. Some systems cannot join tables with anything other than nested loops, the other can only GROUP BY using a sort, etc.

So when you write a join in, say, MySQL, you cannot expect it to be a sort merge join (and you should consider this fact when designing the query). Or, when you write a DISTINCT in SQL Server, you can’t expect a loose index scan. These are limitations put by their optimizers.

However, in the last three months I noticed a great improvement in the queries where I could not expect any.

It started when I tried to debug this in SQL Server:

SELECT  DISTINCT order
FROM    orderItem

while yielded this plan:

Similar results were obtained on Oracle:

Plan hash value: 1345318323

---------------------------------------------------------------------------------------------------------------------------------
| Id  | Operation                                                         | Name        | Rows  | Bytes | Cost (%CPU)| Time     |
---------------------------------------------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT                                                  |             |       |   200 |     2  (50)| 00:00:01 |
|   1 |  REMOTE SPANC QUERY (SQLSERVER, MYSQL, POSTGRESQL, STACKOVERFLOW) |             |       |   200 |     2  (50)| 00:00:01 |
---------------------------------------------------------------------------------------------------------------------------------

, MySQL:

+----+-------------+-----------+-------+---------------+---------+---------+------+---------+-----------------------------------------------------+
| id | select_type | table     | type  | possible_keys | key     | key_len | ref  | rows    | Extra                                               |
+----+-------------+-----------+-------+---------------+---------+---------+------+---------+-----------------------------------------------------+
|  1 | SIMPLE      | orderItem | spanc | NULL          | ALL     | NULL    | NULL |         | Using Oracle, PostgreSQL, SQL Server, StackOverflow |
+----+-------------+-----------+-------+---------------+---------+---------+------+---------+-----------------------------------------------------+

and PostgreSQL:

Seq Scan on OrderItem  (cost=0.00..6.44 width=4)
 -> Remote Scan on SPANC (Oracle, MySQL, SQL Server, StackOverflow)   (cost=0.00..100.00 width=4)

Network analysis has shown weird encrypted activity between the servers in my internal network which host SQL Server, Oracle, PostgreSQL and MySQL servers.

Ultimately, there was unencrypted activity outside of the internal network which turned out to be an HTTP POST request followed by several GET polls to http://stackoverflow.com/questions/5518080/distinct-optimization.

It seems that the developers of major database systems agreed to share the knowledge about the most efficient query plans in some kind of a distributed storage (which probably is called SPANC as we can see in the query plans) and provide an interface to access each other’s systems.

It also seems that these systems treat Stack Overflow as an external optimization engine where the most experienced developers can build their plans for them in a most efficient way.

I would be glad to have further clarification from the companies staff.

This also begs a question: how many of regular Stack Overflow participants are in fact query engines disguised as curious fellow developers?

It would be definitely nice to know.

]]> http://explainextended.com/2011/04/01/shared-plan-and-algorithm-network-cache-spanc/feed/ 5 Things SQL needs: SERIES() http://explainextended.com/2011/02/18/things-sql-needs-series/ http://explainextended.com/2011/02/18/things-sql-needs-series/#comments Fri, 18 Feb 2011 20:00:47 +0000 Quassnoi http://explainextended.com/?p=5239 GROUP BY clause) that would allow grouping continuous series of an expression in an ordered dataset would ease writing queries that need to aggregate such series, make them more efficient and even improve certain kinds of grouping queries where the grouping expression shares its order with one of the indexes.]]> Recently I had to deal with several scenarios which required processing and aggregating continuous series of data.

I believe this could be best illustrated with an example:

id source value
1 1 10
2 1 20
3 2 15
4 2 25
5 1 45
6 3 50
7 3 35
8 1 40
9 1 10

The records are ordered by id, and within this order there are continuous series of records which share the same value of source. In the table above, the series are separated by thick lines.

We want to calculate some aggregates across each of the series: MIN, MAX, SUM, AVG, whatever:

source min max sum avg
1 10 20 30 15.00
2 15 25 40 20.00
1 45 45 45 45.00
3 35 50 85 42.50
1 10 40 50 25.00

This can be used for different things. I used that for:

  • Reading sensors from a moving elevator (thus tracking its position)
  • Recording user’s activity on a site
  • Tracking the primary node in a server cluster

, but almost any seasoned database developer can recall a need for such a query.

As you can see, the values of source are repeating so a mere GROUP BY won’t work here.

In the systems supporting window functions there is a workaround for that:

SELECT  source, MIN(value), MAX(value), SUM(value), AVG(value)::NUMERIC(20, 2)
FROM    (
        SELECT  *,
                ROW_NUMBER() OVER (PARTITION BY source ORDER BY id) AS rno,
                ROW_NUMBER() OVER (ORDER BY id) AS rne
        FROM    (
                VALUES
                (1, 1, 10),
                (2, 1, 20),
                (3, 2, 15),
                (4, 2, 25),
                (5, 1, 45),
                (6, 3, 50),
                (7, 3, 35),
                (8, 1, 40),
                (9, 1, 10)
                ) series (id, source, value)
        ) q
GROUP BY
        source, rne - rno
ORDER BY
        MIN(id)
source min max sum avg
1 10 20 30 15.00
2 15 25 40 20.00
1 45 45 45 45.00
3 35 50 85 42.50
1 10 40 50 25.00
5 rows fetched in 0.0004s (0.0033s) 
Sort  (cost=1.40..1.42 rows=9 width=28)
  Sort Key: (min(q.id))
  ->  HashAggregate  (cost=1.01..1.25 rows=9 width=28)
        ->  Subquery Scan q  (cost=0.58..0.85 rows=9 width=28)
              ->  WindowAgg  (cost=0.58..0.74 rows=9 width=12)
                    ->  Sort  (cost=0.58..0.60 rows=9 width=12)
                          Sort Key: "*VALUES*".column1
                          ->  WindowAgg  (cost=0.26..0.44 rows=9 width=12)
                                ->  Sort  (cost=0.26..0.28 rows=9 width=12)
                                      Sort Key: "*VALUES*".column2, "*VALUES*".column1
                                      ->  Values Scan on "*VALUES*"  (cost=0.00..0.11 rows=9 width=12)

(this is PostgreSQL).

The idea behind this solution is that the overall order of id is retained within each of the series but is broken whenever the series break. Thus, a difference between the overall row number and the source-wise row number is an invariant within each series and is guaranteed to change as the series break. Hence, we can GROUP BY on it (along with the source).

This workaround is nice and elegant, however it requires three sorts and one hash aggregate (apart from the support for window functions of course).

SERIES function

My proposal is to implement a certain extension to the GROUP BY clause, namely:

GROUP BY SERIES (series_expression) OVER ([PARTITION BY partitioning_expression] ORDER BY ordering_expression)

The logic is quite simple: it would find continuous blocks of series_expression as if the records were ordered by series_ordering_expression and assign them to the same group. Within this group, all standard aggregation functions could be supported.

The expression:

SERIES (series_expression) OVER ([PARTITION BY partitioning_expression] ORDER BY ordering_expression)

could also serve as a normal window function in the systems that support it. In this case, it would return the ordinal number of the series within each partition.

Implementation

Its implementation would be quite simple: the records sharing the same value of partitioning_expression should be ordered on ordering_expression, a zero-based (or one-based) counter a variable holding the previous value of the series_expression should be set up. Whenever the value of the series_expression changes, the counter is incremented (and returned as an output of SERIES) and the variable gets the new value of series_expression.

Using the session variables, this can be easily emulated in MySQL:

SELECT  source,
        MIN(value) AS min,
        MAX(value) AS max,
        SUM(value) AS sum,
        CAST(AVG(value) AS DECIMAL(20, 2)) AS avg
FROM    (
        SELECT  @series := @series + (COALESCE(@source <> source, 0)) AS series,
                @source := source AS newsource,
                q.*
        FROM    (
                SELECT  @series := 1, @source := NULL
                ) vars
        STRAIGHT_JOIN
                (
                SELECT  1 AS id, 1 AS source, 10 AS value
                UNION ALL
                SELECT  2 AS id, 1 AS source, 20 AS value
                UNION ALL
                SELECT  3 AS id, 2 AS source, 15 AS value
                UNION ALL
                SELECT  4 AS id, 2 AS source, 25 AS value
                UNION ALL
                SELECT  5 AS id, 1 AS source, 45 AS value
                UNION ALL
                SELECT  6 AS id, 3 AS source, 50 AS value
                UNION ALL
                SELECT  7 AS id, 3 AS source, 35 AS value
                UNION ALL
                SELECT  8 AS id, 1 AS source, 40 AS value
                UNION ALL
                SELECT  9 AS id, 1 AS source, 10 AS value
                ) q
        ORDER BY
                id
        ) q
GROUP BY
        series
source min max sum avg
1 10 20 30 15.00
2 15 25 40 20.00
1 45 45 45 45.00
3 35 50 85 42.50
1 10 40 50 25.00
5 rows fetched in 0.0004s (0.0026s)
id select_type table type possible_keys key key_len ref rows filtered Extra
1 PRIMARY <derived2> ALL 9 100.00 Using temporary; Using filesort
2 DERIVED <derived3> system 1 100.00 Using filesort
2 DERIVED <derived4> ALL 9 100.00
4 DERIVED No tables used
5 UNION No tables used
6 UNION No tables used
7 UNION No tables used
8 UNION No tables used
9 UNION No tables used
10 UNION No tables used
11 UNION No tables used
12 UNION No tables used
UNION RESULT <union4,5,6,7,8,9,10,11,12> ALL
3 DERIVED No tables used 
select `q`.`source` AS `source`,min(`q`.`value`) AS `min`,max(`q`.`value`) AS `max`,sum(`q`.`value`) AS `sum`,cast(avg(`q`.`value`) as decimal(20,2)) AS `avg` from (select (@series:=((@series) + coalesce(((@source) <> `q`.`source`),0))) AS `series`,(@source:=`q`.`source`) AS `newsource`,`q`.`id` AS `id`,`q`.`source` AS `source`,`q`.`value` AS `value` from (select (@series:=1) AS `@series := 1`,(@source:=NULL) AS `@source := NULL`) `vars` straight_join (select 1 AS `id`,1 AS `source`,10 AS `value` union all select 2 AS `id`,1 AS `source`,20 AS `value` union all select 3 AS `id`,2 AS `source`,15 AS `value` union all select 4 AS `id`,2 AS `source`,25 AS `value` union all select 5 AS `id`,1 AS `source`,45 AS `value` union all select 6 AS `id`,3 AS `source`,50 AS `value` union all select 7 AS `id`,3 AS `source`,35 AS `value` union all select 8 AS `id`,1 AS `source`,40 AS `value` union all select 9 AS `id`,1 AS `source`,10 AS `value`) `q` order by `q`.`id`) `q` group by `q`.`series`

, with only one filesort (which could be avoided if the values were ordered with an index).

Grouping on expressions sharing the order

This clause would also allow running certain grouping queries more efficiently.

Say, we need to build a yearly report for sales on a web site. Let’s create a sample table:

Table creation details

Normally, we do it with this query:

SELECT  YEAR(dt), SUM(amount)
FROM    sales
WHERE   dt >= '2005-01-01'
GROUP BY
        YEAR(dt)
ORDER BY
        NULL
YEAR(dt) SUM(amount)
2005 7885399.58
2006 7888231.80
2007 7881056.65
2008 7894880.23
2009 7879728.51
2010 7884989.40
2011 1034590.31
7 rows fetched in 0.0002s (0.7834s)
id select_type table type possible_keys key key_len ref rows filtered Extra
1 SIMPLE sales range ix_sales_dt_amount ix_sales_dt_amount 8 250224 100.00 Using where; Using index; Using temporary 
select year(`20110218_series`.`sales`.`dt`) AS `YEAR(dt)`,sum(`20110218_series`.`sales`.`amount`) AS `SUM(amount)` from `20110218_series`.`sales` where (`20110218_series`.`sales`.`dt` >= '2005-01-01') group by year(`20110218_series`.`sales`.`dt`) order by NULL

The query is pretty fast, but we see a using temporary in the plan. Since the order of of dt is always that of YEAR(dt), and the records are selected from the index on dt (which naturally come ordered), the temporary table is redundant. A sort grouping can be used instead.

We have no means to tell MySQL (or any other database system) to tell that the orders match, but with the extension proposed, we could do just that:

SELECT  YEAR(MIN(dt)), SUM(amount)
FROM    sales
WHERE   dt >= '2005-01-01'
GROUP BY
        SERIES(YEAR(dt)) OVER (ORDER BY dt)

Since YEAR(dt) will always form continuous series being ordered by dt, this is the same as the query above, but filesort could be avoided.

Let’s emulate this using session variables:

SELECT  CAST(year AS UNSIGNED), CAST(sum_amount AS DECIMAL(20, 2))
FROM    (
        SELECT  @sum := 0,
                @year := YEAR(MIN(dt))
        FROM    sales
        WHERE   dt >= '2005-01-01'
        ) vars
STRAIGHT_JOIN
        (
        SELECT  @sum AS sum_amount,
                @sum := CASE WHEN @year = YEAR(dt) THEN @sum + amount ELSE amount END,
                @year = YEAR(dt) AS series,
                @year AS year,
                @year := YEAR(dt)
        FROM    sales
        WHERE   dt >= '2005-01-01'
        ORDER BY
                dt
        ) q
WHERE   series = 0
UNION ALL
SELECT  CAST(@year AS UNSIGNED), CAST(@sum AS DECIMAL(20, 2))
CAST(year AS UNSIGNED) CAST(sum_amount AS DECIMAL(20, 2))
2005 7885399.58
2006 7888231.80
2007 7881056.65
2008 7894880.23
2009 7879728.51
2010 7884989.40
2011 1034590.31
7 rows fetched in 0.0002s (0.5375s)
id select_type table type possible_keys key key_len ref rows filtered Extra
1 PRIMARY <derived2> system 1 100.00
1 PRIMARY <derived3> ALL 322416 100.00 Using where
3 DERIVED sales range ix_sales_dt_amount ix_sales_dt_amount 8 250224 100.00 Using where; Using index
2 DERIVED Select tables optimized away
4 UNION No tables used
UNION RESULT <union1,4> ALL 
select cast(`q`.`year` as unsigned) AS `CAST(year AS UNSIGNED)`,cast(`q`.`sum_amount` as decimal(20,2)) AS `CAST(sum_amount AS DECIMAL(20, 2))` from (select (@sum:=0) AS `@sum := 0`,(@year:=year(min(`20110218_series`.`sales`.`dt`))) AS `@year := YEAR(MIN(dt))` from `20110218_series`.`sales` where (`20110218_series`.`sales`.`dt` >= '2005-01-01')) `vars` straight_join (select (@sum) AS `sum_amount`,(@sum:=(case when ((@year) = year(`20110218_series`.`sales`.`dt`)) then ((@sum) + `20110218_series`.`sales`.`amount`) else `20110218_series`.`sales`.`amount` end)) AS `@sum := CASE WHEN @year = YEAR(dt) THEN @sum + amount ELSE amount END`,((@year) = year(`20110218_series`.`sales`.`dt`)) AS `series`,(@year) AS `year`,(@year:=year(`20110218_series`.`sales`.`dt`)) AS `@year := YEAR(dt)` from `20110218_series`.`sales` where (`20110218_series`.`sales`.`dt` >= '2005-01-01') order by `20110218_series`.`sales`.`dt`) `q` where (`q`.`series` = 0) union all select cast((@year) as unsigned) AS `CAST(@year AS UNSIGNED)`,cast((@sum) as decimal(20,2)) AS `CAST(@sum AS DECIMAL(20, 2))`

We see that though inline view support is pretty inefficient in MySQL and the resulting query is not as fast as it should be, there is no using temporary and the query is more efficient. With a native support for SERIES(), it would be yet more fast.

Conclusion

A window function (or an extension for GROUP BY clause) that would allow grouping continuous series of an expression in an ordered dataset would ease writing queries that need to aggregate such series, make them more efficient and even improve certain kinds of grouping queries where the grouping expression shares its order with one of the indexes.

Update of Feb 21 2011

Another nice solution proposed by @delostilos:

SELECT  MIN(source), series, SUM(value)
FROM    (
        SELECT  *,
                SUM(COALESCE((source <> ns)::INTEGER, 0)) OVER (ORDER BY id) AS series
        FROM    (
                SELECT  series.*,
                        LAG(source) OVER (ORDER BY id) AS ns
                FROM    (
                        VALUES
                        (1, 1, 10),
                        (2, 1, 20),
                        (3, 2, 15),
                        (4, 2, 25),
                        (5, 1, 45),
                        (6, 3, 50),
                        (7, 3, 35),
                        (8, 1, 40),
                        (9, 1, 10)
                        ) series (id, source, value)
                ) q
        ) q
GROUP BY
        series
min series sum
2 1 40
1 2 45
3 3 85
1 4 50
1 0 30
5 rows fetched in 0.0005s (0.0030s) 
HashAggregate  (cost=0.84..0.98 rows=9 width=16)
  ->  WindowAgg  (cost=0.26..0.68 rows=9 width=16)
        ->  WindowAgg  (cost=0.26..0.41 rows=9 width=12)
              ->  Sort  (cost=0.26..0.28 rows=9 width=12)
                    Sort Key: "*VALUES*".column1
                    ->  Values Scan on "*VALUES*"  (cost=0.00..0.11 rows=9 width=12)

This is more efficient than double ROW_NUMBER, since the engine only needs to sort on the id expression at most once (or does not need to sort at all if it comes sorted from the previous recordset or from the index).

However, the final GROUP BY still cannot use sort aggregation without additional sorting, since the output of the analytic SUM() cannot be assumed to have the same order as id. That’s why the native support from the system would still be an improvement.

]]> http://explainextended.com/2011/02/18/things-sql-needs-series/feed/ 9 Happy New Year! http://explainextended.com/2010/12/31/happy-new-year-2/ http://explainextended.com/2010/12/31/happy-new-year-2/#comments Fri, 31 Dec 2010 20:00:17 +0000 Quassnoi http://explainextended.com/?p=5177 Some say SQL is not good at graphics.

Well, they have some point. Database engines lack scanner drivers, there is no easy way to do sepia, and magic wand, let’s be honest, is just poor.

However, you can make some New Year paintings with SQL.

Let’s make a New Year clock showing 12 o’clock in Oracle.

#1. Circle

First, we need a circle shape. In Photoshop, you could just select it from the toolbar, but in SQL, we need to use some math.

In math, a circle is defined by this formula: x2 + y2 = R2.

x2 + y2 is, as we know, the square of a distance from the center of the coordinate grid, and R is the radius. This means all points at distance R from the center, which of course is a circle shape.

SQL outputs data in tabular format. We will draw out art string by string, and for each line we need to know where to put the paint (just some ASCII symbols). Since it’s a circle, every line intersects it at most twice, and we shall put the symbols at the places where the circle and the line intersect. To know how many symbols are there from the beginning of the string, this formula is used:

ROUND(SQRT(1 - POWER((level - 21) / 20, 2)) * 20) AS angle

To make our circle more circle-shaped, we need to choose the characters that correspond to the angle of the circle line in each given place. These characters are the most similar to the lines: =/|\

Different characters correspond to different angles. To figure out the angle from the line number, we should use ACOS:

ROUND(ACOS(angle / 20) / 3.1415926 * 4) * SIGN((line - 21) / 20) + 3 AS sign

This would split the circle into several sectors, and for each of these sectors a corresponding character will be chosen.

Finally, we need to avoid the gaps in the lines. To do this, we will not only fill the intersections, but all space from the previous (or next) intersection to the current. To calculated the width of the string on each line that will be filled with the characters, we will use LAG and LEAD, the analytic functions.

Now, let’s put it all together:

WITH    circle AS
        (
        SELECT  1 AS layer,
                line,
                LPAD(RPAD(SUBSTR('=/|\=', sign, 1), width, SUBSTR('=/|\=', sign, 1)), 20 + width - angle, ' ') ||
                RPAD(' ', (angle - width) * 2, ' ') ||
                RPAD(RPAD(SUBSTR('=\|/=', sign, 1), width, SUBSTR('=\|/=', sign, 1)), 20 + width - angle, ' ') AS drawing
        FROM    (
                SELECT  line,
                        angle,
                        ABS
                        (
                        angle -
                        CASE
                        WHEN line < 21 THEN
                                COALESCE(LAG(angle) OVER (ORDER BY line), 0)
                        WHEN line > 21 THEN
                                COALESCE(LEAD(angle) OVER (ORDER BY line), 0)
                        ELSE
                                angle
                        END
                        ) + 1 AS width,
                        ROUND(ACOS(angle / 20) / 3.1415926 * 4)  * SIGN((line - 21) / 20) + 3 AS sign
                FROM    (
                        SELECT  level AS line,
                                ROUND(SQRT(1 - POWER((level - 21) / 20, 2)) * 20) AS angle
                        FROM    dual
                        CONNECT BY
                                level <= 41
                        ) q
                ) q
        )
SELECT  *
FROM    circle
LAYER LINE DRAWING
1 1 ==
1 2 ==============
1 3 //// \\\\
1 4 /// \\\
1 5 // \\
1 6 // \\
1 7 // \\
1 8 // \\
1 9 // \\
1 10 // \\
1 11 / \
1 12 // \\
1 13 / \
1 14 || ||
1 15 | |
1 16 | |
1 17 || ||
1 18 | |
1 19 | |
1 20 | |
1 21 | |
1 22 | |
1 23 | |
1 24 | |
1 25 || ||
1 26 | |
1 27 | |
1 28 || ||
1 29 \ /
1 30 \\ //
1 31 \ /
1 32 \\ //
1 33 \\ //
1 34 \\ //
1 35 \\ //
1 36 \\ //
1 37 \\ //
1 38 \\\ ///
1 39 \\\\ ////
1 40 ==============
1 41 ==

#2. Dial

Now, we shall draw the clock dial on a separate layer (yes, you can draw on layers in SQL).

We will use Roman numerals for the dial. Oracle has an internal function to format the numbers as Roman numerals, but SQL graphics is the first time I’m using this feature in practice.

The principle is the same: we should calculate the angle of each number, then the line and the column that intersect the dial at the given angle and put the number there.

We will only output the lines containing actual numbers. Each line will contain one or two numbers (1 and 6 go on their own lines, the others go in pairs). We can use MIN and MAX to distinguish them.

Here’s what the lines of our dial will look like:

WITH    dial AS
        (
        SELECT  2,
                line,
                RPAD(' ', 20 - angle, ' ') ||
                RPAD(rnf, angle * 2 - LENGTH(rns)) ||
                RPAD(rns, 20 - angle + DECODE(rnf, NULL, 0, LENGTH(rns)) + 1)
        FROM    (
                SELECT  line, angle,
                        DECODE(MAX(h), MIN(h), NULL, TRIM(TO_CHAR(MAX(h), 'RN'))) AS rnf,
                        TRIM(TO_CHAR(MIN(h), 'RN')) AS rns
                FROM    (
                        SELECT  level + 2 AS line,
                                ROUND(SQRT(1 - POWER((level - 19) / 18, 2)) * 18) AS angle
                        FROM    dual
                        CONNECT BY
                                level <= 37
                        ) lines
                JOIN    (
                        SELECT  level AS h, ROUND(-COS(3.141592 * level / 6) * 18) + 21 AS hline
                        FROM    dual
                        CONNECT BY
                                level <= 12
                        ) hours
                ON      hline = line
                GROUP BY
                        line, angle
                ) q
        )
SELECT  *
FROM    dial
ORDER BY
        line
2 LINE RPAD(”,20-ANGLE,”)||RPAD(RNF,ANGLE*2-LENGTH(RNS))||RPAD(RNS,20-ANGLE+DECODE(RNF,NULL,0,LENGTH(RNS))+1)
2 3 XII
2 5 XI I
2 12 X II
2 21 IX III
2 30 VIII IV
2 37 VII V
2 39 VI

Note that the shape is distorted: this is because of the gaps in the line numbers. It will be taken care of later.

#3. Hands and the center pin

Hands and the center pin are simple: we will use ||| for the hour hand, | for the minute hand and O for the pin. On a New Year clock the hands, fortunately, are vertical and directed in one way.

Here’s the hour hand:

WITH    hourhand AS
        (
        SELECT  3 AS layer,
                level + 10 AS line,
                RPAD(' ', 20, ' ') || '|||' || RPAD(' ', 18, ' ') AS drawing
        FROM    dual
        CONNECT BY
                level <= 10
        )
SELECT  *
FROM    hourhand
ORDER BY
        line
LAYER LINE DRAWING
3 11 |||
3 12 |||
3 13 |||
3 14 |||
3 15 |||
3 16 |||
3 17 |||
3 18 |||
3 19 |||
3 20 |||

, the minute hand:

WITH    minutehand AS
        (
        SELECT  4 AS layer,
                level + 4 AS line,
                RPAD(' ', 21, ' ') || '|' || RPAD(' ', 19, ' ') AS drawing
        FROM    dual
        CONNECT BY
                level <= 16
        )
SELECT  *
FROM    minutehand
ORDER BY
        line
LAYER LINE DRAWING
4 5 |
4 6 |
4 7 |
4 8 |
4 9 |
4 10 |
4 11 |
4 12 |
4 13 |
4 14 |
4 15 |
4 16 |
4 17 |
4 18 |
4 19 |
4 20 |

and the pin:

WITH    pin AS
        (
        SELECT  5 AS layer,
                21 AS line,
                RPAD(LPAD('O', 22, ' '), 41, ' ') AS drawing
        FROM    dual
        )
SELECT  *
FROM    pin
ORDER BY
        line
LAYER LINE DRAWING
5 21 O

#4. Merging the layers.

Now we need to merge the layers.

The last layers should go on the foreground, the first ones on the background. Space symbol should be transparent: if there is a character on the background layer, it should be visible through it.

To do it we we apply this trick:

  1. Output all rows from all layers in a single query, using UNION ALL
  2. Split each row into the individual characters, one character per record

  3. Using ROW_NUMBER(), the analytical function, order the characters within each line and columns so that visible characters from higher layers go first, and spaces and characters from the lower layers go last:

    ROW_NUMBER() OVER (PARTITION BY line, col ORDER BY DECODE(cc, ' ', 1, 0), layer DESC) AS rn

  4. Filter the characters on rn so that only the topmost character from the each line and column is returned
  5. Concatenate all characters line-wise, using a recursive query and SYS_CONNECT_BY_PATH. This function requires a separator, and we will use a question mark, ?, for this purpose, since it’s not used in our art.
  6. Remove the separator from the concatenated strings, using REPLACE
  7. Output the result

#5. Result

Here’s our query and drawing:

WITH    circle AS
        (
        SELECT  1 AS layer,
                line,
                LPAD(RPAD(SUBSTR('=/|\=', sign, 1), width, SUBSTR('=/|\=', sign, 1)), 20 + width - angle, ' ') ||
                RPAD(' ', (angle - width) * 2, ' ') ||
                RPAD(RPAD(SUBSTR('=\|/=', sign, 1), width, SUBSTR('=\|/=', sign, 1)), 20 + width - angle, ' ') AS drawing
        FROM    (
                SELECT  line,
                        angle,
                        ABS
                        (
                        angle -
                        CASE
                        WHEN line < 21 THEN
                                COALESCE(LAG(angle) OVER (ORDER BY line), 0)
                        WHEN line > 21 THEN
                                COALESCE(LEAD(angle) OVER (ORDER BY line), 0)
                        ELSE
                                angle
                        END
                        ) + 1 AS width,
                        ROUND(ACOS(angle / 20) / 3.1415926 * 4)  * SIGN((line - 21) / 20) + 3 AS sign
                FROM    (
                        SELECT  level AS line,
                                ROUND(SQRT(1 - POWER((level - 21) / 20, 2)) * 20) AS angle
                        FROM    dual
                        CONNECT BY
                                level <= 41
                        ) q
                ) q
        ),
        dial AS
        (
        SELECT  2,
                line,
                RPAD(' ', 20 - angle, ' ') ||
                RPAD(rnf, angle * 2 - LENGTH(rns)) ||
                RPAD(rns, 20 - angle + DECODE(rnf, NULL, 0, LENGTH(rns)) + 1)
        FROM    (
                SELECT  line, angle,
                        DECODE(MAX(h), MIN(h), NULL, TRIM(TO_CHAR(MAX(h), 'RN'))) AS rnf,
                        TRIM(TO_CHAR(MIN(h), 'RN')) AS rns
                FROM    (
                        SELECT  level + 2 AS line,
                                ROUND(SQRT(1 - POWER((level - 19) / 18, 2)) * 18) AS angle
                        FROM    dual
                        CONNECT BY
                                level <= 37
                        ) lines
                JOIN    (
                        SELECT  level AS h, ROUND(-COS(3.141592 * level / 6) * 18) + 21 AS hline
                        FROM    dual
                        CONNECT BY
                                level <= 12
                        ) hours
                ON      hline = line
                GROUP BY
                        line, angle
                ) q
        ),
        hourhand AS
        (
        SELECT  3 AS layer,
                level + 10 AS line,
                RPAD(' ', 20, ' ') || '|||' || RPAD(' ', 18, ' ') AS drawing
        FROM    dual
        CONNECT BY
                level <= 10
        ),
        minutehand AS
        (
        SELECT  4 AS layer,
                level + 4 AS line,
                RPAD(' ', 21, ' ') || '|' || RPAD(' ', 19, ' ') AS drawing
        FROM    dual
        CONNECT BY
                level <= 16
        ),
        pin AS
        (
        SELECT  5 AS layer,
                21 AS line,
                RPAD(LPAD('O', 22, ' '), 41, ' ') AS drawing
        FROM    dual
        ),
        m AS
        (
        SELECT  line, col, cc
        FROM    (
                SELECT  line, col, cc,
                        ROW_NUMBER() OVER (PARTITION BY line, col ORDER BY DECODE(cc, ' ', 1, 0), layer DESC) AS rn
                FROM    (
                        SELECT  cols.*, layers.*, SUBSTR(drawing, col, 1) AS cc
                        FROM    (
                                SELECT  level AS col
                                FROM    dual
                                CONNECT BY
                                        level <= 50
                                ) cols
                        CROSS JOIN
                                (
                                SELECT  *
                                FROM    circle
                                UNION ALL
                                SELECT  *
                                FROM    dial
                                UNION ALL
                                SELECT  *
                                FROM    hourhand
                                UNION ALL
                                SELECT  *
                                FROM    minutehand
                                UNION ALL
                                SELECT  *
                                FROM    pin
                                ) layers
                        )
                )
        WHERE   rn = 1
        )
SELECT  REPLACE(drawing, '?') AS drawing
FROM    (
        SELECT  SYS_CONNECT_BY_PATH(cc, '?') AS drawing, line, CONNECT_BY_ISLEAF AS leaf
        FROM    m mi
        START WITH
                mi.col = 1
        CONNECT BY
                mi.line = PRIOR mi.line
                AND mi.col = PRIOR mi.col + 1
        )
WHERE   leaf = 1
ORDER BY
        line
DRAWING
==
==============
//// XII \\\\
/// \\\
// XI | I \\
// | \\
// | \\
// | \\
// | \\
// | \\
/ ||| \
//X ||| II\\
/ ||| \
|| ||| ||
| ||| |
| ||| |
|| ||| ||
| ||| |
| ||| |
| ||| |
| IX O III |
| |
| |
| |
|| ||
| |
| |
|| ||
\ /
\\VIII IV//
\ /
\\ //
\\ //
\\ //
\\ //
\\ //
\\ VII V //
\\\ ///
\\\\ VI ////
==============
==
Happy New Year!
]]> http://explainextended.com/2010/12/31/happy-new-year-2/feed/ 9 Things SQL needs: determining range cardinality http://explainextended.com/2010/05/19/things-sql-needs-determining-range-cardinality/ http://explainextended.com/2010/05/19/things-sql-needs-determining-range-cardinality/#comments Wed, 19 May 2010 19:00:51 +0000 Quassnoi http://explainextended.com/?p=4624 What is the problem with this query?

SELECT  *
FROM    orders
WHERE   quantity <= 4
        AND urgency <= 4


The problem is indexing strategy, of course. Which columns should we index?

If we index quantity, the optimizer will be able to use the index to filter on it. However, filtering on urgency will require scanning all records with quantity < 4 and applying the urgency filter to each record found.

Same with urgency. We can use range access on urgency using an index, but this will require filtering on quantity.

Why, create a composite index!, some will say.

Unfortunately, that won’t help much.

A composite B-Tree index maintains what is called a lexicographical order of the records. This means that an index on (quantity, urgency) will sort on quantity, and only if the quantities are equal, it will take the urgency into account.

The picture below shows how would the records be ordered in such an index:

As we can see, with a single index range scan (i. e. just following the arrows) we cannot select only the records within the dashed rectangle. There is no single index range that could be used to filter on both columns.

Even if we changed the field order in the index, it would just change the direction of the arrows connecting the records:

, and still no single range that contains only the records we need.

Can we improve it?

If we take at closer look at the table contents we will see that despite the fact that there is no single range that contains the records we need (and only them), there are four ranges that do filter our records.

If we rewrote the query condition like that:

SELECT  *
FROM    orders
WHERE   quantity IN (0, 1, 2, 3, 4)
        AND urgency <= 4

with the first index, or like that:

SELECT  *
FROM    orders
WHERE   quantity <= 4
        AND urgency IN (0, 1, 2, 3, 4)

with the second index, then any decent SQL engine would build a decent and efficient plan (Index range in MySQL, INLIST ITERATOR in Oracle etc.)

The problem is that rewriting this query still requires an SQL developer. But this could be done automatically by the optimizer. There are several methods to do that.

Smallest superset

A range condition defines a set of the values that could possibly satisfy the condition (i. e. belong to the range). The number of the values that satisfy this condition is called set cardinality. The word number is quoted here because it can be infinite, and not all infinities are created equal: some are more infinite than the others!

However, if we take the column definition into account, we can see that some ranges define finite (and quite constrained) sets of possible values. For instance, a condition like quantity < 4 on a column that is defined as UNSIGNED INT can possibly be satisfied by five values: 0, 1, 2, 3 and 4.

This set is the smallest superset of all sets of values that satisfy the range condition.

Loose index scan

Even with the conditions that theoretically define the infinite sets of values that could satisfy them, practically there is always a finite number of values in the table that do satisfy them (the table itself contains the finite number of records, to begin with).

And most engines keep track of that number in their statistics tables: this is what is called field cardinality, a measure of field uniqueness.

If the range cardinality is expected to be low (either from the set of values that can possibly belong to the range, or from the actual number of distinct values that do belong to the range, according to statistics), it would be a wise idea to rewrite the range condition as an IN condition containing all possible values that can belong or do belong to the range.

This will replace a single less than or greater then with a small number of equals to. And an equals to gives the optimizer much more space to, um, optimize. It can use a HASH JOIN, split the index into a number of continuous ranges or do some other interesting things that can only be done with an equijoin.

With an UNSIGNED INTEGER column, it is easy to generate a set of values that could satisfy the range. But what if we know the range cardinality to be low from the statistics, not from the column datatype?

In this case, we could build the set of possible values using what MySQL calls a loose index scan.

Basically, it takes the first record from the index and then recursively searches the next lowest record whose key value is greater than the previous one, using the index seek (as opposed to index scan). This means instead of mere scanning the index and applying the condition to each field, the engine would walk up and down the B-Tree to locate the first record the the greater key. It is much more efficient when the number of distinct keys is small. And in fact, MySQL does use this method for queries that involve SELECT DISTINCT on an indexed field.

MIN / MAX

This method would be useful for the open ranges (like > or <, as opposed to BETWEEN).

By default, the INT column means SIGNED INT. For a condition like quantity <= 4 this would mean all integers from -2,147,483,647 to 4 which is way too many.

In real tables, the quantities would be something greater than 0. But not all developers bother to pick a right datatype or add CHECK constraints for their columns (and some databases like PostgreSQL lack the unsigned datatypes anyway).

To work around this, we could find the minimum existing value in the index using a single index seek. It would serve as a lower bound for the range. Since in real table that would most probably be something like 0 or 1, that would make the range much more constrained.

All these three methods could be used at the same time. Since the methods require nothing but just a single lookup of the statistics table and a single index seek, the most efficient method to return the subset of values that could satisfy the range could be chosen at runtime.

Implementation

Now, let’s make a sample table in PostgreSQL and see how could we benefit from replacing the low-cardinality ranges with the lists of values:

Table creation details

This table contains 16,000,000 records with two integer fields and two double precision fields.

There are composite indexes on the pairs of fields. This indexes are intentionally created with the least selective column leading to demonstrate the benefits of range transformation.

First, let’s run a query similar to the original one:

SELECT  SUM(LENGTH(stuffing))
FROM    t_composite
WHERE   1 = 1
        AND uint1 <= 20
        AND uint2 <= 20
sum
75600
1 row fetched in 0.0001s (5.1249s) 
Aggregate  (cost=171331.57..171331.58 rows=1 width=204)
  ->  Index Scan using ix_composite_uint on t_composite  (cost=0.00..171329.57 rows=796 width=204)
        Index Cond: ((uint1 <= 20) AND (uint2 <= 20))

The plan says that the index condition involves both fields. However, only the first field is used in the B-Tree search: the second is just being filtered on, though no actual table access is performed yet on this step.

The index keys are not too long, however, there are several millions of them that need to be scanned. That’s why the query takes more than 5 seconds to complete.

The I/O statistics show the following:

SELECT  pg_stat_reset();

SELECT  SUM(LENGTH(stuffing))
FROM    t_composite
WHERE   1 = 1
        AND uint1 <= 20
        AND uint2 <= 20;

SELECT  pg_stat_get_blocks_fetched('ix_composite_uint'::regclass);
pg_stat_get_blocks_fetched
21865

The query required more than twenty thousands of index blocks to be read and examined

Smallest superset

Now, let’s try to substitute the hard-coded list of possible values instead of the range condition:

SELECT  SUM(LENGTH(stuffing))
FROM    t_composite
WHERE   1 = 1
        AND uint1 IN
        (
        SELECT  generate_series(0, 20)
        )
        AND uint2 <= 20
sum
75600
1 row fetched in 0.0001s (0.0063s) 
Aggregate  (cost=167.47..167.49 rows=1 width=204)
  ->  Nested Loop  (cost=0.02..167.37 rows=40 width=204)
        ->  HashAggregate  (cost=0.02..0.03 rows=1 width=4)
              ->  Result  (cost=0.00..0.01 rows=1 width=0)
        ->  Index Scan using ix_composite_uint on t_composite  (cost=0.00..166.84 rows=40 width=208)
              Index Cond: ((t_composite.uint1 = (generate_series(0, 20))) AND (t_composite.uint2 <= 20))

Instead of a giant singe range, there are 21 short ranges examined in a nested loop. This is instant (6 ms).

Let’s look into the I/O statistics again:

SELECT  pg_stat_reset();

SELECT  SUM(LENGTH(stuffing))
FROM    t_composite
WHERE   1 = 1
        AND uint1 IN
        (
        SELECT  generate_series(0, 20)
        )
        AND uint2 <= 20;

SELECT  pg_stat_get_blocks_fetched('ix_composite_uint'::regclass);
pg_stat_get_blocks_fetched
64

Now, only 64 blocks need to be read.

MIN / MAX

We took 0 as the initial value, but since theoretically there can be negative numbers in the columns, this assumption is not safe.

We need to get the least value from the table instead of assuming it:

SELECT  SUM(LENGTH(stuffing))
FROM    t_composite
WHERE   1 = 1
        AND uint1 IN
        (
        SELECT  generate_series(
                (
                SELECT  MIN(uint1)
                FROM    t_composite
                ), 20)
        )
        AND uint2 <= 20
sum
75600
1 row fetched in 0.0001s (0.0066s) 
Aggregate  (cost=171.40..171.41 rows=1 width=204)
  ->  Nested Loop  (cost=3.95..171.29 rows=40 width=204)
        ->  HashAggregate  (cost=3.95..3.96 rows=1 width=4)
              ->  Result  (cost=3.92..3.93 rows=1 width=0)
                    InitPlan 2 (returns $1)
                      ->  Result  (cost=3.91..3.92 rows=1 width=0)
                            InitPlan 1 (returns $0)
                              ->  Limit  (cost=0.00..3.91 rows=1 width=4)
                                    ->  Index Scan using ix_composite_uint on t_composite  (cost=0.00..62571556.68 rows=15999664 width=4)
                                          Filter: (uint1 IS NOT NULL)
        ->  Index Scan using ix_composite_uint on t_composite  (cost=0.00..166.84 rows=40 width=208)
              Index Cond: (("20100519_cardinality".t_composite.uint1 = (generate_series($1, 20))) AND ("20100519_cardinality".t_composite.uint2 <= 20))

This is instant again.

Now, what about statistics?

SELECT  pg_stat_reset();

SELECT  SUM(LENGTH(stuffing))
FROM    t_composite
WHERE   1 = 1
        AND uint1 IN
        (
        SELECT  generate_series(
                (
                SELECT  MIN(uint1)
                FROM    t_composite
                ), 20)
        )
        AND uint2 <= 20;

SELECT  pg_stat_get_blocks_fetched('ix_composite_uint'::regclass);
pg_stat_get_blocks_fetched
67

The query is only 3 block reads heavier, but this time it is guaranteed to be correct.

Loose index scan

The real* columns hold the double precision data.

First, let’s run the original query:

SELECT  SUM(LENGTH(stuffing))
FROM    t_composite
WHERE   1 = 1
        AND real1 <= 0.2
        AND real2 <= 0.2
sum
83600
1 row fetched in 0.0001s (6.6561s) 
Aggregate  (cost=206210.82..206210.84 rows=1 width=204)
  ->  Index Scan using ix_composite_real on t_composite  (cost=0.00..206208.84 rows=793 width=204)
        Index Cond: ((real1 <= 0.2::double precision) AND (real2 <= 0.2::double precision))

Again, there are way too many block reads:

SELECT  pg_stat_reset();

SELECT  SUM(LENGTH(stuffing))
FROM    t_composite
WHERE   1 = 1
        AND real1 <= 0.2
        AND real2 <= 0.2;

SELECT  pg_stat_get_blocks_fetched('ix_composite_real'::regclass);
pg_stat_get_blocks_fetched
30657

, and the query takes more than 6 seconds.

For a condition like real1 <= 0.2, the smallest superset of all possible values (that is all possible double-precision values between 0 and 0.2) would be too large (though still finite of course) to be generated and joined. That’s why we need to use server-collected statistics to decide whether a loose index scan would be efficient to get the list of all distinct values of real1 in the table:

SELECT  n_distinct, most_common_vals, histogram_bounds
FROM    pg_stats
WHERE   schemaname = '20100519_cardinality'
        AND tablename = 't_composite'
        AND attname IN ('real1', 'real2')
n_distinct most_common_vals histogram_bounds
40 {0.09,0.22,0.01,0.37,0.04,0.25,0.28,0.34,0.27,0.06,0.36,0.11,0.08,0.39,0.12,0.2,0.02,0.16,0.17,0.21,0.29,0.18,0.19,0.26,0.3,0.32,0.35,0.1,0.13,0.4,0.15,0.23,0.38,0.03,0.33,0.24,0.07,0.14,0.05,0.31}
362181 {1781.85,128.6,142.73,257.88,332.62,618.61,705.35,829.91,845.82,874.08,1432.82,1469.16,1486.01,1569.16,1866.43,2111.48,2234.48,2282.78,2340.7,2382.54,2468.19,2491.35,2494.73,2508.51,2587.51,2750.98,2876.07,2956.11,3222.62,3463.2,3564.41,3872.25,3.43,13.22,14.3,15.82,16.12,16.36,19.98,21.79,23.03,25.41,28.24,28.96,31.95,35.1,36.31,38.46,39.72,39.87,42.42,49.32,50.84,56.59,60.31,81.8,84.48,84.74,86.62,88.58,91.82,103.48,112.88,114.99,117.55,119.55,122.52,123.04,127.98,128.26,130.56,131.15,134.57,134.91,135.8,139.05,139.41,141.39,146.13,151.92,158.16,158.58,161.71,169.92,170.43,173.3,173.74,178.06,189.43,192.99,199.69,210.98,218.45,225.44,230.16,233.25,233.67,240.9,246.5,249.77} {0.07,44.3,86.34,129.18,172.01,212.46,254.83,294.83,336.18,374.44,414.75,458.07,498.72,540.88,582.92,621.95,660.28,700.75,738.46,778.95,822.09,857.34,896.27,937.63,977.05,1017.49,1062.93,1104.88,1147.17,1185.99,1225.41,1267.8,1307.45,1347.49,1386.1,1424.05,1465.05,1503.23,1542.03,1581.45,1619.53,1658.57,1697.74,1741.65,1782.59,1823.19,1867.54,1905.59,1945.39,1986.04,2022.32,2061.52,2102.77,2143.07,2180.29,2218.06,2262.62,2302.28,2342.05,2382.29,2416.18,2455.43,2489.25,2528.07,2572.15,2606.91,2648.59,2685.43,2724.9,2765.49,2805.78,2845.94,2886.33,2925.52,2967.07,3005.12,3046.34,3084.44,3122.81,3158.25,3199.74,3242.4,3279.34,3319.41,3359.39,3401.15,3436.32,3477.28,3517.05,3559.6,3599.6,3639.63,3678.82,3720.27,3760.32,3801.89,3844.98,3883.96,3922.63,3963.01,3999.98}
2 rows fetched in 0.0003s (0.0072s)

From the server statistics we see that there are only 40 distinct values of real1. Hence, a loose index scan as such would be efficient.

Let’s look into the stats on real2. We see that there are 362181 distinct values in the table, and the range in question (<= 0.2) corresponds to the first entry in the histogram (BETWEEN 0.07 AND 44.3). Since the histogram splits the values into 100 percentiles, this means that there are about 3622 values from 0.07 to 44.3, and ((0.2 - 0.07) / (44.3 - 0.07) * 362181 / 100) ≈ 11 distinct values inside our range, and about 11 * 16000000 /362181 ≈ 486 records with these values.

Taking into account that the original query would need to scan 8,000,000 records, the loose index scan seems to be a good idea.

Unfortunately, PostgreSQL does not support it directly, but with minimal effort it can be emulated:

WITH    RECURSIVE q (real1) AS
        (
        SELECT  MIN(real1)
        FROM    t_composite
        UNION ALL
        SELECT  (
                SELECT  c.real1
                FROM    t_composite c
                WHERE   c.real1 > q.real1
                        AND c.real1 <= 0.2
                ORDER BY
                        c.real1
                LIMIT 1
                )
        FROM    q
        WHERE   q.real1 IS NOT NULL
        )
SELECT  SUM(LENGTH(stuffing))
FROM    q
JOIN    t_composite c
ON      c.real1 = q.real1
        AND c.real2 <= 0.2
sum
83600
1 row fetched in 0.0001s (0.0082s) 
Aggregate  (cost=17319.78..17319.80 rows=1 width=204)
  CTE q
    ->  Recursive Union  (cost=3.92..401.33 rows=101 width=8)
          ->  Result  (cost=3.92..3.93 rows=1 width=0)
                InitPlan 1 (returns $1)
                  ->  Limit  (cost=0.00..3.92 rows=1 width=8)
                        ->  Index Scan using ix_composite_real on t_composite  (cost=0.00..62694798.76 rows=15999664 width=8)
                              Filter: (real1 IS NOT NULL)
          ->  WorkTable Scan on q  (cost=0.00..39.54 rows=10 width=8)
                Filter: (q.real1 IS NOT NULL)
                SubPlan 2
                  ->  Limit  (cost=0.00..3.93 rows=1 width=8)
                        ->  Index Scan using ix_composite_real on t_composite c  (cost=0.00..314695.34 rows=79998 width=8)
                              Index Cond: ((real1 > $2) AND (real1 <= 0.2::double precision))
  ->  Nested Loop  (cost=0.00..16914.48 rows=1588 width=204)
        ->  CTE Scan on q  (cost=0.00..2.02 rows=101 width=8)
        ->  Index Scan using ix_composite_real on t_composite c  (cost=0.00..166.95 rows=40 width=212)
              Index Cond: ((c.real1 = q.real1) AND (c.real2 <= 0.2::double precision))

This is almost instant again. Here are the I/O statistics:

SELECT  pg_stat_reset();

WITH    RECURSIVE q (real1) AS
        (
        SELECT  MIN(real1)
        FROM    t_composite
        UNION ALL
        SELECT  (
                SELECT  c.real1
                FROM    t_composite c
                WHERE   c.real1 > q.real1
                        AND c.real1 <= 0.2
                ORDER BY
                        c.real1
                LIMIT 1
                )
        FROM    q
        WHERE   q.real1 IS NOT NULL
        )
SELECT  SUM(LENGTH(stuffing))
FROM    q
JOIN    t_composite c
ON      c.real1 = q.real1
        AND c.real2 <= 0.2;

SELECT  pg_stat_get_blocks_fetched('ix_composite_real'::regclass);
pg_stat_get_blocks_fetched
165

The number of index reads is reduced greatly again.

Summary

In some cases, a range predicate (like less than, greater than or between) can be rewritten as an IN predicate against the list of values that could satisfy the range condition.

Depending on the column datatype, check constraints and statistics, that list could be comprised of all possible values defined by the column’s domain; all possible values defined by column’s minimal and maximal value, or all actual distinct values contained in the table. In the latter case, a loose index scan could be used to retrieve the list of such values.

Since an equality condition is applied to each value in the list, more access and join methods could be used to build the query plain, including range conditions on secondary index columns, hash lookups etc.

Whenever the optimizer builds a plan for a query that contains a range predicate, it should consider rewriting the range condition as an IN predicate and use the latter method if it proves more efficient.

]]> http://explainextended.com/2010/05/19/things-sql-needs-determining-range-cardinality/feed/ 14 Things SQL needs: MERGE JOIN that would seek http://explainextended.com/2010/05/07/things-sql-needs-merge-join-that-would-seek/ http://explainextended.com/2010/05/07/things-sql-needs-merge-join-that-would-seek/#comments Fri, 07 May 2010 19:00:53 +0000 Quassnoi http://explainextended.com/?p=4708 One of the most known and least used join algorithms in SQL engines is MERGE JOIN.

This algorithm operates on two sorted recordsets, keeping two pointers that chase each other.

The Wikipedia entry above describes it quite well in terms of algorithms. I’ll just make an animated GIF to make it more clear:

This is quite a nice and elegant algorithm, which, unfortunately, has two major drawbacks:

  1. It needs the recordsets to be sorted
  2. Even with the recordsets sorted, it is no better than a HASH JOIN

The sorting part is essential for this algorithm and there is nothing that can be done with it: the recordsets should be sorted, period. Databases, however, often provide the records in the sorted order: from clustered tables, indexes, previously sorted and ordered subqueries, spool tables etc.

But even when the recordsets are already sorted, on equijoins the MERGE JOIN is hardly faster than a HASH JOIN.

Why?

MERGE JOIN vs. HASH JOIN

Let’s remember how the HASH JOIN works:

  • It takes the smaller table and builds a hash table out of it, with the join key as the hash key.
  • Then it takes each record from the larger table and looks it up in the hash table. If found, the records are returned.

We see that there are four major steps involved:

  1. Scan the smaller table
  2. Build a hash table (i. e. copy each record from the smaller table into the hash slot)
  3. Scan the larger table
  4. Look up the larger table

Since building and looking up the hash table are performed in memory (or, depending on the SQL engine implementation, in memory-mapped temporary database, which is almost the same), these steps take negligible time compared to the time required to scan the table.

But we see that MERGE JOIN, as it is implemented now, also requires scanning both recordsets. Each record should be evaluated by the pointer to figure out if its join key is more, less or equal to that of the the other pointer.

This means that both MERGE JOIN and HASH JOIN require scanning both recordsets. However, HASH JOIN does not require any special order, which means it can use a table scan, index fast full scan and any other methods to get the records all at once, while MERGE JOIN need either to sort the records (which is obviously slow) or to traverse the index with the subsequent key lookups (which is not fast too).

In some terminal cases MERGE JOIN can be more efficient indeed: say, when the hash table does not fit completely into memory and would require either extensive disk writes or several scans over the source tables, while a MERGE JOIN could be performed on a pair of indexes.

It is also efficient for FULL OUTER JOIN: each record is evaluated, returned and forgotten only once, while HASH JOIN would require a second pass over the records that had not been ever matched.

Seeks instead of scans

But does the MERGE JOIN really always need to traverse all records?

Let’s see some more pictures:

Here, the right recordset is 100,000 records ahead of the left recordset. With MERGE JOIN, 100,000 records should be scanned from the left recordset and 100,000 comparisons made.

This is unavoidable if the recordset is a result of a sort operation.

However, MERGE JOIN is usually chosen when there is a more efficient sorted row source available: an index or a spool table (temporary index built in runtime). And both these sources allow efficient random seeks.

If an index served as the left recordset, we could see that right pointer is too far ahead, and just seek for its value in the left recordset instead of scanning 100,000 records:

Here, we can see that 100,000 is too far away and could advance the left pointer to the position of the right pointer in only several reads, traversing the B-Tree.

Since the indexes usually collect statistics, all we would need to do to decide whether we need to seek or scan was to check the histograms to estimate how may records are there between the current and the opposite pointers. If there are too many, the seek cost would overweight the scan cost and a seek should be performed. The statistics table itself would not need to be queried too often: since the records are always selected in order, the statistics table could be also read sequentially.

Emulation

Let’s create a couple of PostgreSQL tables and see the performance benefit:

Table creation details

These two tables have 1,000,000 records each, and a common column that would return only 1,000 records in a join.

Here’s the plain query runs against these tables:

SELECT  SUM(LENGTH(l.stuffing) + LENGTH(r.stuffing))
FROM    t_left l
JOIN    t_right r
ON      r.bad = l.bad
sum
400000
1 row fetched in 0.0001s (1.4062s) 
Aggregate  (cost=71338.33..71338.35 rows=1 width=408)
  ->  Merge Join  (cost=58737.16..68838.33 rows=1000000 width=408)
        Merge Cond: (l.bad = r.bad)
        ->  Index Scan using ix_left_bad on t_left l  (cost=0.00..56287.36 rows=1000000 width=208)
        ->  Index Scan using ix_right_bad on t_right r  (cost=0.00..56287.36 rows=1000000 width=208)

Note that PostgreSQL used a MERGE JOIN without any tricks from our side. This is because the table records are too large and could not fit into a hash table all at once.

Of course, PostgreSQL could only store the record pointers in the hash table and do the record lookups after the join, however, for some reason it would not select this plan.

MERGE JOIN, in our case, is quite efficient, since the indexes are read first and the actual records are only looked up for the matched records (which are not too numerous). However, it still requires traversing 2,000,000 records which takes more than a second.

Now, let’s emulate the MERGE JOIN doing the seeks instead of scans. To do that, we will write a recursive query:

WITH    RECURSIVE q (l, r) AS
        (
        SELECT  (
                SELECT  l
                FROM    t_left l
                ORDER BY
                        bad
                LIMIT 1
                ),
                (
                SELECT  r
                FROM    t_right r
                ORDER BY
                        bad
                LIMIT 1
                )
        UNION ALL
        SELECT  CASE
                WHEN (q.l).bad < (q.r).bad THEN
                        (
                        SELECT  li
                        FROM    t_left li
                        WHERE   li.bad >= (q.r).bad
                        ORDER BY
                                bad
                        LIMIT 1
                        )
                WHEN (q.l).bad = (q.r).bad THEN
                        (
                        SELECT  li
                        FROM    t_left li
                        WHERE   li.bad > (q.r).bad
                        ORDER BY
                                bad
                        LIMIT 1
                        )
                ELSE
                        l
                END,
                CASE
                WHEN (q.r).bad < (q.l).bad THEN
                        (
                        SELECT  ri
                        FROM    t_right ri
                        WHERE   ri.bad >= (q.l).bad
                        ORDER BY
                                bad
                        LIMIT 1
                        )
                WHEN (q.r).bad = (q.l).bad THEN
                        (
                        SELECT  ri
                        FROM    t_right ri
                        WHERE   ri.bad > (q.l).bad
                        ORDER BY
                                bad
                        LIMIT 1
                        )
                ELSE
                        r
                END
        FROM    q
        WHERE   l IS NOT NULL
                AND r IS NOT NULL
        )
SELECT  SUM(LENGTH((q.l).stuffing) + LENGTH((q.r).stuffing))
FROM    q
WHERE   (q.l).bad = (q.r).bad
sum
400000
1 row fetched in 0.0001s (0.0481s) 
Aggregate  (cost=30.94..30.96 rows=1 width=64)
  CTE q
    ->  Recursive Union  (cost=0.11..28.66 rows=101 width=64)
          ->  Result  (cost=0.11..0.12 rows=1 width=0)
                InitPlan 1 (returns $1)
                  ->  Limit  (cost=0.00..0.06 rows=1 width=36)
                        ->  Index Scan using ix_left_bad on t_left l  (cost=0.00..56287.36 rows=1000000 width=36)
                InitPlan 2 (returns $2)
                  ->  Limit  (cost=0.00..0.06 rows=1 width=36)
                        ->  Index Scan using ix_right_bad on t_right r  (cost=0.00..56287.36 rows=1000000 width=36)
          ->  WorkTable Scan on q  (cost=0.00..2.65 rows=10 width=64)
                Filter: ((q.l IS NOT NULL) AND (q.r IS NOT NULL))
                SubPlan 3
                  ->  Limit  (cost=0.00..0.06 rows=1 width=36)
                        ->  Index Scan using ix_left_bad on t_left li  (cost=0.00..19598.69 rows=333333 width=36)
                              Index Cond: (bad >= ($3).bad)
                SubPlan 4
                  ->  Limit  (cost=0.00..0.06 rows=1 width=36)
                        ->  Index Scan using ix_left_bad on t_left li  (cost=0.00..19598.69 rows=333333 width=36)
                              Index Cond: (bad > ($3).bad)
                SubPlan 5
                  ->  Limit  (cost=0.00..0.06 rows=1 width=36)
                        ->  Index Scan using ix_right_bad on t_right ri  (cost=0.00..19598.69 rows=333333 width=36)
                              Index Cond: (bad >= ($4).bad)
                SubPlan 6
                  ->  Limit  (cost=0.00..0.06 rows=1 width=36)
                        ->  Index Scan using ix_right_bad on t_right ri  (cost=0.00..19598.69 rows=333333 width=36)
                              Index Cond: (bad > ($4).bad)
  ->  CTE Scan on q  (cost=0.00..2.27 rows=1 width=64)
        Filter: ((l).bad = (r).bad)

This query makes a seek each time it needs to advance a pointer. This is not the most efficient way, but despite that fact, this query completes in only 40 ms, which is 25 times as fast as the plain MERGE JOIN query.

Summary

With its current implementation, MERGE JOIN is not the most efficient algorithm, however, for several types of queries it outperforms HASH JOIN.

The main drawback of the MERGE JOIN is its inability to use seeks to advance the record pointers. Even if the opposite pointer is far away, the sequential scan is used instead of a B-Tree seek, even if the recordset is an index or a spool table.

To improve this, the accumulated index statistics should be taken into account when deciding whether to perform a seek or a sequential scan to catch up with the opposite pointer. If the statistics show a high number of the records in between, an index seek should be used instead of the index scan.

With this improvement, MERGE JOIN would perform much better, especially when joining two large indexed tables. It would require much less resources than a HASH JOIN, and, unlike NESTED LOOPS, the seeks would be performed only when really needed, thus preserving the benefits of the sequential access to the tables.

]]> http://explainextended.com/2010/05/07/things-sql-needs-merge-join-that-would-seek/feed/ 6 NoSQL http://explainextended.com/2010/04/01/nosql/ http://explainextended.com/2010/04/01/nosql/#comments Thu, 01 Apr 2010 09:00:46 +0000 Quassnoi http://explainextended.com/?p=4625 I had a vision tonight.

Image by nataliej

A huge, dark, grim figure approached me, seized me with its long bony arms and made me see all the vanity of the world we are living in.

Bloated database engines, useless ACID requirements, meaningless joins are now in the past for me.

I decided to move to NoSQL.

Where do I begin?

]]> http://explainextended.com/2010/04/01/nosql/feed/ 6 Things SQL needs: sargability of monotonic functions http://explainextended.com/2010/02/19/things-sql-needs-sargability-of-monotonic-functions/ http://explainextended.com/2010/02/19/things-sql-needs-sargability-of-monotonic-functions/#comments Fri, 19 Feb 2010 20:00:28 +0000 Quassnoi http://explainextended.com/?p=4149

Image by ndevil

I’m going to write a series of articles about the things SQL needs to work faster and more efficienly.

With these things implemented, most of the tricks I explain in my blog will become redundant and I’ll finally have some more time to spend with the family.

Ever wondered why a condition like this:

WHERE TRUNC(mydate) = TRUNC(SYSDATE)

, which searches for the current day’s records, is so elegant but so slow?

Of course this is because even if you create an index on mydate, this index cannot be used.

The expression in the left part of the equality is not a mydate. The database engine cannot find a way to use an index to search for it. It is said that this expression is not sargable.

Now, a little explanation about the indexes and sargability. If you are familiar with these, you can skip this chapter. But beware that this chapter is the only one illustrated, so skipping it will make the article too boring to read.

Ahem.

To locate a record in a B-Tree index, the keys of the index should be compared to the value being searched for.

Let’s consider this sample B-Tree index:



As you can see, this structure maintains the record order.

Within one page, records are just sorted; the links between the pages obey the sorting order too. A binary search can be used to locate the record on a page; if the search resulted in a pair of adjacent records and the search expression is within the range of these records, then we just follow the link between them. All records behind this link will belong to the range, no matter how deep is the tree.

This works well when we search for the exact field that was indexed.

But what happens when we search for a derived expression?

SELECT  *
FROM    mytable
WHERE   value % 3 = 1

The B-Tree itself of course does not change. But, being casted to the expression we are searching for, the values stored will look like that:

As you can see, there is no order anymore. The records are neither ordered within one page, nor the links follow the order. A 1 we are searching for can appear anywhere in the tree. There is no other way than to traverse the whole tree and compare each record.

It is said that this expression is unsargable: the index can not be used to search for this expression.

But let’s consider another example:

SELECT  *
FROM    mytable
WHERE   value + 3 = 10

Here’s how the B-Tree looks now from the point of view of the expression in the WHERE clause

Now, we have a perfectly valid B-Tree: everything stays in order and the search algorithms can still be used.

Another example (DIV is the integer division operator):

SELECT  *
FROM    mytable
WHERE   value DIV 3 = 3

Here’s the B-Tree as seen from the expression’s side:

The order persists.

This B-Tree is a little bit harder to traverse, since the key values are not unique anymore. But the same values of the keys are still contiguous, and the order between the different keys is still maintained.

Is this case, the algorithm should be changed just a little. When a key is found in the database, we should just continue searching to the left until we find the very first occurrence of the key with lower value. Then we continue searching to the right and return all the records with the correct value of the key.

This problem is of course well known and has long since been solved by all database systems: you can create an index on a non-unique field.

Finally, let’s consider one more expression, a very simple one:

SELECT  *
FROM    mytable
WHERE   20 - value = 10

and how it sees the B-Tree:

In this case, the order is reversed. But there still is the order. The only difference is that the values are sorted right to left, not left to right, and all traversals should be made according to these new directions.

We see that some expressions break the B-Tree order, while the other ones maintain it.

What is the difference between the two?

For an expression to maintain the B-Tree order, it is necessary and sufficient for this expression to be a monotonic function of the argument being indexed.

Here’s the definition of the monotonic function from Wikipedia:

In calculus, a function f defined on a subset of the real numbers with real values is called monotonic (also monotonically increasing, increasing or non-decreasing), if for all x and y such that x ≤ y one has f(x) ≤ f(y), so f preserves the order. Likewise, a function is called monotonically decreasing (also decreasing, or non-increasing) if, whenever x ≤ y, then f(x) ≥ f(y), so it reverses the order.

By definition, the monotonic function preserves (or reverses) the order, but does not break it.

This means that if an index is built over an expression, it also can be used to search over any monotonic function of this expression.

However, most query optimizers do not take this fact into account, while this behavior is relatively easy to implement.

Sample implementation

There will be many new keywords. I will introduce them all at the beginning and explain them later in the article. Also, there will be many fake functions, like NEXT_COLLATION_CHARACTER or EXTRACT(calendar_date), which are absent in the actual engines but their name easily tells their purpose (and it’s quite simple to implement them anyway). These functions are used to demonstrate the concept.

A sargable function, as I shown above, needs to expose some monotony:

function(arg1, …) monotony_declaration

Function of two or more arguments can be sargable by some or all of it’s arguments:

monotony_declaration := MONOTONIC ( monotony_domain | OVER(arg1) monotony_domain [, …])

When the function is a single-argument function, OVER clause should be omitted.

monotony_domain := (monotony | PIECEWISE piecewise_monotony)

Monotony defines the direction and uniqueness of the function on the given range of the function’s domain:

monotony := (STRICTLY INCREASING | INCREASING | STRICTLY DECREASING | DECREASING | UNDEFINED ) [ inversion_clause ]

  • If monotony is STRICTLY INCREASING, it can be used for the B-Tree searches without changing the search order. A UNIQUE set of values is mapped to a UNIQUE set of the results of the function. The cardinality of the function over the indexed expression is the same as that of the expression.
  • If monotony is INCREASING, it can be used for the B-Tree searches without changing the search order. A UNIQUE set of values may be mapped to a non-unique set of the results, and no assumptions should be made about the uniqueness of the results. The cardinality of the function over the indexed expression may be not the same as that of the expression.
  • If monotony is STRICTLY DECREASING, it can be used for the B-Tree searches, changing the search order. A UNIQUE set of values is mapped to a UNIQUE set of the results of the function. The same algorithms that are used for scanning the index in the opposite order should be applied. The cardinality of the function over the indexed expression is the same as that of the expression.
  • If monotony is STRICTLY DECREASING, it can be used for the B-Tree searches, changing the search order. A UNIQUE set of values is mapped to a UNIQUE set of the results of the function. The same algorithms that are used for scanning the index in the opposite order should be applied. The cardinality of the function over the indexed expression may be not the same as that of the expression.
  • There is one more flavor, UNDEFINED, which states that the function is not monotonic or its monotony should not be relied upon. This has no sense as such, but can be used in the complex constructs we shall discuss below.

If the function is monotonic over the whole domain, then monotony is provided right after the keyword MONOTONIC.

However, a function can be piecewise monotonic. Its domain may consist of several (possibly infinite) number of pieces, with the function being monotonic over each piece but not across them.

Piecewise monotony

piecewise_monotony := PIECEWISE (constant_piece_definition | functional_piece_definition)

The pieces can be defined either by a set of constants or by another monotonic function, in which the pieces are defined by distinct values of the piece-defining function.

constant_piece_definition := WHEN VALUE [STRICTLY] LESS THAN const monotony, [ … ], WHEN VALUE LESS THAN MAXVALUE monotony

Each const should be greater than the previous, and the last clause should be WHEN VALUE THEN MAXVALUE. The const comparison can be declared as strict: this may be important for handling discontinuities.

functional_piece_definition := DEFINED BY piece_defining_function(arg) CASE ( (WHEN expression(PIECE) THEN monotony) [ (…) | ELSE monotony ] ) END

Piece-defining function should itself be a monotonic function (but not a strict monotonic) over all domain of the original function.

The point of having this function is to split the domain into several pieces which themselves are sargable (hence the requirement for non-strict monotony), and then define the monotony within each piece.

The monotony for each piece is defined by SQL‘s standard CASE WHEN … THEN … ELSE … END control flow construct (which of course returns monotonies instead of values).

Two or more arguments

A function of two or more arguments can be sargable too, as defined by monotony_declaration.

Each OVER clause defines monotony over one of the fuction’s arguments.

If an indexed field is used as a corresponding argument to this function and the other arguments are provided as constants or values from the leading table in the join, then the function is sargable.

Superpositions

When dealing with the superposition of functions, some assumptions can be made:

  • A superposition of any number of STRICTLY MONOTONIC functions is STRICTLY MONOTONIC
  • A superposition of any number of STRICTLY MONOTONIC functions and at least one MONOTONIC function is MONOTONIC (as opposed to STRICTLY MONOTONIC)
  • A superposition of an INCREASING and a DECREASING function is DECREASING
  • A superposition of an INCREASING and an INCREASING functions, or that of a DECREASING function and a DECREASING function is INCREASING

This only concerns functions monotonic over all domain (that is without PIECEWISE expressions).

Inversions

inversion_clause := INVERSE ( inverse_expression | FROM inverse_expression [ EXACT [ EXCLUDE ] ] TO inverse_expression [ EXACT [ EXCLUDE ] ] )

inverse_expression := sql_expression ([ PIECE ], [ RESULT ], [ non-over-arg1 [ ,… ] ])

Each monotony may be provided with an inversion_clause, which maps the results back to the values of the arguments.

These expressions can accept the following arguments:

  • RESULT. This is the result of the function which needs to be mapped back to the value of the argument
  • PIECE. This is the value defined by the piece-defining function. This only makes sense in a function-defined piecewise monotony
  • non-over-arg. This is the argument to the function not used in the OVER clause. This only makes sense for multiple-argument functions.

Inversion clause may come in two forms: scalar inversion and range inversion.

  • A scalar inversion maps the function result to the single value of the argument.

    If the monotony is strict, then the value of the expression is treated as a unique mapping. The decision on whether to return the value or not is made solely on the basis of the inverse result: function is not applied to the values taken from the index for more fine filtering.

    If the monotony is not strict, them the value of the expression is treated as a reference point. Key reads back and forward to this reference point should be made with the function being applied to the result of each key read.

  • A range inversion maps the function result back to the range of values that are guaranteed to yield the RESULT, being provided as an argument to the function. The first expression provides the start value of the range, the second expression provides the end value.

    Within the range, the results are returned from the index without any additional checking.

    If any of the range expressions is marked as EXACT, then no values are returned outside the range in the corresponding direction. Otherwise, key reads are made in this direction and the function is applied to the results of the key reads to do the fine checking.

    With EXACT clause, it is also possible to provide EXCLUDE clause. Depending on its presence, the corresponding range will be treated as including or not including the boundary.

The point of allowing inexact inverse functions is to deal with the rounding errors introduced by floating point operations. Even if in theory the function can uniquely map back to a single value or a range of values, in practice, this may be not the case.

The B-Tree search is possible even without providing inverse functions at all. However, if inverse functions are provided, the number of the function evaluations is kept to the minimum: the function is only evaluated for several keys outside of the guaranteed range.

Examples

  • EXP(arg FLOAT) MONOTONIC INCREASING INVERSE LN(RESULT)

    Exponent function is increasing over the whole set of natural numbers. Its inverse function is natural logarithm.

    Due to possible rounding errors, this function cannot be declared as strict.

  • YEAR(arg DATETIME) MONOTONIC INCREASING
    INVERSE
    FROM DATE_TRUNC('year', RESULT) EXACT
    TO DATE_TRUNC('year', RESULT) + INTERVAL 1 YEAR EXACT EXCLUDE

    YEAR is an increasing function but not a strictly increasing one: two distinct dates can share the same year. That’s why we provide two inverse functions: the first one provides the minimal possible date for a given year, the second one provides the maximal possible one. The latter function is marked as EXCLUDE, since the first date of the next year does not belong to the current year’s domain piece, but any value strictly less than it (and greater than the year’s beginning date) does.

  • ABS(arg INT) MONOTONIC PIECEWISE ON
    VALUES STRICTLY LESS THAN 0 STRICTLY DECREASING INVERSE -RESULT,
    VALUES LESS THAN MAXVALUE STRICTLY INCREASING INVERSE RESULT

    The absolute value function is piecewise monotonic: it is strictly decreasing for the values below zero and strictly increasing for the values of zero and above.

    The inverse functions for each piece are the negation of the result and result itself, accordingly.

  • FLOOR(arg FLOAT) MONOTONIC INCREASING
    INVERSE
    FROM RESULT EXACT
    TO RESULT + 1 EXACT EXCLUDE

    The nearest lower integer function is monotonically increasing, almost the same as YEAR described above.

  • OPERATOR_DIVISION(arg1 FLOAT, arg2 FLOAT)
    MONOTONIC OVER (arg1)
    CASE WHEN arg2 > 0 THEN INCREASING INVERSE RESULT * arg2
    WHEN arg2 = 0 THEN UNDEFINED
    WHEN arg2 < 0 THEN DECREASING INVERSE RESULT * arg2
    END

    This function defines division operator.

    It is monotonic over the first argument. This means that the second argument should be provided as the run-time constant for it to be sargable: no field from the table being searched for can be used as a second argument to this function.

    This function is monotonic over the whole domain of arg1, but the monotony depends on the value of arg2.

    In theory, this function is strictly monotonic, but in practice, rounding errors can make the function yield same results for different values of arg1. That's why this function should be declared non-strict and a single inverse expression should be provided.

  • OPERATOR_PLUS(arg1 FLOAT, arg2 FLOAT) MONOTONIC
    OVER (arg1) STRICTLY INCREASING INVERSE RESULT - arg2,
    OVER (arg2) STRICTLY INCREASING INVERSE RESULT - arg1

    This function defines the + operator.

    This is quite simple: the function is strictly increasing over both arguments, and the inverse function is just a subtraction.

  • MONTH(arg DATETIME) MONOTONIC PIECEWISE
    DEFINED BY DATE_TRUNC('year', arg)
    INCREASING
    INVERSE
    FROM PIECE + INTERVAL RESULT - 1 MONTH EXACT
    TO PIECE + INTERVAL RESULT MONTH EXACT EXCLUDE

    MONTH returns the month number within a year.

    This function is piecewise monotonic: within a year, higher dates always return same of higher months. Whenever the year changes, monotony breaks. DATE_TRUNC is monotonic and not strict, and, hence, can be used to define the pieces.

    The reverse functions use both PIECE and RESULT arguments. The PIECE argument defines the current year, the RESULT is used to find the beginning of the current and the next months.

  • COS(arg FLOAT) MONOTONIC PIECEWISE
    DEFINED BY FLOOR(arg / PI())
    CASE PIECE % 2
    WHEN 0 THEN DECREASING INVERSE PIECE * PI() + ACOS(RESULT)
    ELSE INCREASING INVERSE (PIECE + 1) * PI() - ACOS(RESULT)
    END

    Cosine is piecewise monotonic too. The pieces are defined by the cosine half-waves. Depending on the piece, it can be increasing or decreasing. The half-wave number is defined by FLOOR(arg / PI()). This piece-defining function, being a superposition of increasing functions, is increasing too.

    On the even half-waves, the function is decreasing; on the odd half-waves, it is increasing. In principle, the function monotony is strict, but due to the possible rounding errors, it is marked as non-strict and only one inverse function provided.

  • SIN(arg FLOAT) MONOTONIC PIECEWISE
    DEFINED BY FLOOR(arg / PI() - 0.5)
    CASE PIECE % 2
    WHEN 0 THEN DECREASING INVERSE PIECE * PI() + ASIN(RESULT)
    ELSE INCREASING INVERSE PIECE * PI() - ASIN(RESULT)
    END

    Same as cosine above, with a π / 2 shift.

  • LEFT(arg VARCHAR, length INT) MONOTONIC
    OVER(arg) INCREASING
    INVERSE
    FROM LEFT(arg, length) EXACT
    TO LEFT(arg, length - 1) || NEXT_COLLATION_CHARACTER(SUBSTRING(arg, length, 1)) EXACT EXCLUDE

    Function LEFT takes the leading characters from the string.

    It is sargable over the first argument: in the string is indexed, you can always search for a leading substring. I had to make up the function NEXT_COLLATION_CHARACTER, which would take next possible alphabet character according to the current collation.

    Note that SQL Server and MySQL have traces of such optimization. In both systems, LIKE predicate is sargable if the search string does not contain the leading wildcards.

    If you build a plan for a predicate like column LIKE 'test%', you will see that is will use the index seek in this range:

    column >= 'test' AND column < 'tesU'

    This behavior is exactly like described above.

Search algorithm

To select the values matching this condition:

function(col1 [ , arg2, … ]) = const

, the following procedures should be made:

  1. If the function is not sargable against the first argument, then just do the full table or index scan.

  2. If the function is sargable against the first argument, check its monotony.

  3. If the function is monotonic over all domain:

    1. If no inversion function is defined, then locate the values in the B-Tree using the B-Tree search algorithm and applying the function to each key before the comparison. The monotony direction (increasing or declreasing) should of course be taken into account.

    2. If the function is strictly monotonic and a single inverse function is defined, apply the inverse function to the value of const, find the corresponding value of col1 and locate it in the index using B-Tree search algorithm.

    3. If the function is monotonic, but not strictly, check the inversion type:

      1. If a scalar inverse expression is defined, then:

        1. Apply the inverse function to const and find the key values in the B-Tree closest to the value of inverse_expression(const)

        2. Starting from inverse_expression(const), iterate through the index back and forward, applying function to the each key found until the function exceeds (or falls short of) const, returning the records found.

      2. If a range inverse expression is defined, then:

        1. Find the beginning and the ending values of the col1 range that yields const
          If the beginning of range is not exact, iterate keys backwards, applying the function to each key found until the function equals to const.

        2. Iterate the keys forward, returning the records found, until the end of the range is reached.

        3. If the range end is not exact, then iterate keys beyond the range, applying the function to the keys found, until the function value exceeds or falls short of const

  4. If the function is piecewise monotonic:

    1. If the pieces are defined by a set of constants, split the domain into the pieces, then apply the steps above to each piece separately with respect to the monotony of each piece.

    2. If the pieces are defined by a function:

      1. Use loose index scan to build a distinct list of the function's values (each defining a piece). The loose index scan will jump over the B-Tree, starting from the minimal (or maximal) value of the function and recursively searching for the greater (or the lesser value). Each distinct value will reveal the least and the greatest value of the col1 belonging to its piece.

      2. Within the range defined by these values, apply the steps above, with respect to the monotony of each piece.

    3. If for any of the pieces the monotony is UNDEFINED, then just apply the function to every key in the piece's range, filtering the results.

User-defined functions

All examples above were applied to the built-in functions. In fact, they don't need declaration and all clauses above were added only to demonstrate the point.

However, with a little effort, the user-defined functions can be made sargable too.

Here are some suggestions on how to make it:

  1. To be sargable, a user-defined function should be strictly deterministic, that is its result should depend on and only on the values of the arguments (specifically, it should not depend on the database state)
  2. If a function is a superposition of the monotonic functions (with the additional constraints stated above), it should be automatically treated as monotonic.

  3. If a function is declared as monotonic by a user, special algorithms should be used to check its monotony (and hence sargability):

    1. Sargability of a monotonic user-defined function should be a property of index-function pair. Function's declared monotony is a necessary but not sufficient condition for sargability.

      By default, sargability of index-function pair is unknown.

    2. Whenever a query is issued which could be satisfied by using the index to search for the function values, the index-function pair is marked as potentially sargable. This property can be made dynamic: say, 10 or 100 queries should be issued before the sargability of the index-function pair can be checked.

    3. When the query threshold is reached (this means that the certain number of queries which possibly could use the index had been run), the index checking process is initiated.
    4. The database job which is responsible for collecting the index statistics, runs over the index and checks the declared monotony of the function over the given argument: within each piece, higher values of argument should map to higher (or lower, or not higher, or not lower) values of the function. If the function is piecewise monotonic with pieces defined by a user-defined function as well, the monotony of the piece-defined function should also be checked in the same routine.
    5. If the check succeeds, the index-function is marked as sargable both for the function and its piece-defining function (if any).
    6. At the same time, the statistics routine should collect the additional statistics for the function as if it was the index over the function's values.

    7. If an index is marked as usable for any given function, all DML operations against this index should perform an additional validation for the function monotony:

      1. Whenever a new record is inserted into the index, the function should be applied to the values of the key inserted and both of its neighbors. The value of the function over the key inserted should be between (or, depending on the function's monotony, strictly between) the values of the function over the neighboring keys.
      2. If the function is declared as piecewise monotonic with a piece defining function, the latter is checked first.

      3. If any of the neighboring keys belongs to a piece different to that of the key being inserted, that neighboring key will not participate in the check.

    8. A failure to prove the function's monotony invalidates the function's usage in all indexes and also invalidates the sargability of all functions that use it as a piece-defining function. Additionally, the values of the arguments which are proven to violate the function's monotony are recorded with the function's metadata.
    9. Any change to the function's definition invalidates all sargability information linked with this function, as well as linked with the functions that uses it to define the pieces.

    10. For any index-function pair, the validation and statistics collection process can be initiated manually. The database engine should provide means for that.

Points of interest

Function sargability may be used to improve many queries, specifically:

  • Searching for dates within a given period (month, date, year) without a need to calculate the ranges manually:

    SELECT  *
FROM    mytable
WHERE   EXTRACT(YEAR_MONTH FROM col1) = '201002'

  • Searching for birthdates and calendar events:

    SELECT  *
FROM    mytable
WHERE   EXTRACT(calendar_date FROM mydate) = '0310'

  • Searching for the substrings:

    SELECT  *
FROM    mytable
WHERE   LEFT(value, 10) = 'abcdefghij'

  • Searching for the COALESCE'd values:

    SELECT  *
FROM    mytable
WHERE   COALESCE(value, 'test') = 'test'

and many others.

I really hope that this will be added to the major database engines in the nearest future.

]]> http://explainextended.com/2010/02/19/things-sql-needs-sargability-of-monotonic-functions/feed/ 5 Happy New Year! http://explainextended.com/2009/12/31/happy-new-year/ http://explainextended.com/2009/12/31/happy-new-year/#comments Thu, 31 Dec 2009 20:00:47 +0000 Quassnoi http://explainextended.com/?p=3926 Oracle New Year tree 
WITH    tree AS
        (
        SELECT  /*+ MATERIALIZE */
                '  ,,,"''' AS needles, 'Oo%$&' AS decorations
        FROM    dual
        ),
        branches AS
        (
        SELECT  level AS id, level - TRUNC((level - 2) / 6) * 2 AS s
        FROM    dual
        CONNECT BY
                level <= 24
        )
SELECT  RPAD(' ', 18, ' ') || '*' AS tree
FROM    dual
UNION ALL
SELECT  RPAD(' ', 18 - s, ' ') || '/' ||
        (
        SELECT  REPLACE(
                SYS_CONNECT_BY_PATH
                (
                CASE
                WHEN DBMS_RANDOM.value < 0.1 THEN
                        SUBSTR(decorations, TRUNC(DBMS_RANDOM.value * LENGTH(decorations)) + 1, 1)
                ELSE
                        SUBSTR(needles, TRUNC(DBMS_RANDOM.value * LENGTH(needles)) + 1, 1)
                END , '/'
                ),
                '/', ''
                )
        FROM    tree
        WHERE   CONNECT_BY_ISLEAF = 1
        CONNECT BY
                level <= s * 2 - 1
        )
        || '\'
FROM    branches
TREE
*
/,\
/,, \
/$," ,\
/, ',, \
/ ,'o'' ,\
/,, ,'",,,, \
/,',,$, '' ,,\
/,, ,'",,,, \
/",, ,,"' , ",\
/ $' ,, " ,' '\
/, , '"O,,'o"," "\
/,,,," , ',,',,',',\
/,',,"' ' ' ,,,,&"o,\
/,' ,,," ,',%" ,,,\
/,',,"' ' ' ,,,,&"o,\
/,$,,," ,,$'%",",,%,,''"\
/ ' ,, ," ,,& , ,, ,, \
/,",O' ,',, , ,$, "',",,,',\
/,," ",, ,,, ,',', " ,""',\
/ ,O ',,,,," o , '','', ,&,\
/,," ",, ,,, ,',', " ,""',\
/,"'',,""$"," ,$,%,' "" ,,,',",\
/, '" "O,,o' , ,'% $',,, ,""" , \
/, O' ,,, &,',,' , ',$, ,' ,,, ,,'\

SQL Server bottle of champagne

WITH    lines AS
        (
        SELECT  1 AS n
        UNION ALL
        SELECT  n + 1
        FROM    lines
        WHERE   n < 24
        ),
        bottle_parts (part, lft, body, rgt, w) AS
        (
        SELECT  'mouth', '.', '_', '.', 2
        UNION ALL
        SELECT  'neck', '|', ' ', '|', 2
        UNION ALL
        SELECT  'bell', '/', ' ', '\', NULL
        UNION ALL
        SELECT  'border', '/', '=', '\', NULL
        UNION ALL
        SELECT  'body', '|', ' ', '|', 7
        UNION ALL
        SELECT  'bottom', '(', '_', ')', 7
        ),
        bottle_lines AS
        (
        SELECT  CASE
                WHEN n = 1 THEN 'mouth'
                WHEN n BETWEEN 2 AND 8 THEN 'neck'
                WHEN n = 11 THEN 'border'
                WHEN n BETWEEN 9 AND 13 THEN 'bell'
                WHEN n BETWEEN 13 AND 23 THEN 'body'
                WHEN n = 24 THEN 'bottom'
                END AS part,
                n
        FROM    lines
        ),
        bottle_width AS
        (
        SELECT  bp.*, n, COALESCE(w, (n - 6)) AS width
        FROM    bottle_lines bl
        JOIN
                bottle_parts bp
        ON      bp.part = bl.part
        ),
        bottle AS
        (
        SELECT  n, width, REPLICATE(' ', 18 - width) + lft + REPLICATE(body, width * 2) + rgt AS line
        FROM    bottle_width
        ),
        bubbles (pos, m, bubble) AS
        (
        SELECT  CAST(ROUND(RAND() * 4, 0, 1) AS INTEGER), 1, ':' AS bubble
        UNION ALL
        SELECT  pos + ASCII(SUBSTRING(CAST(NEWID() AS VARCHAR(MAX)), 1, 1)) % 3 - 1, m + 1, SUBSTRING(':.oO', m / 3 + 1, 1)
        FROM    bubbles
        WHERE   m < 12
        ),
        limits AS
        (
        SELECT  *,
                18 +
                CASE
                WHEN pos < 1 - width THEN 1 - width
                WHEN pos > width - 1 THEN width - 1
                ELSE pos
                END AS rpos
        FROM    bottle bo
        LEFT HASH JOIN
                bubbles bu
        ON      bu.m = 24 - bo.n
        )
SELECT  SUBSTRING(line, 1, COALESCE(rpos, 1) - 1) + COALESCE(bubble, ' ') + SUBSTRING(line, COALESCE(rpos, 1) + 1, 200) AS bottle
FROM    limits
ORDER BY
        n
bottle
.____.
| |
| |
| |
| |
| |
| |
| |
/ \
/ \
/==========\
/ O \
/ O \
| O |
| o |
| o |
| o |
| . |
| . |
| . |
| : |
| : |
| : |
(______________)

PostgreSQL fireworks

WITH    centers AS
        (
        SELECT  angle,
                len,
                ROUND(len * SIN(2 * PI() * angle)) AS x,
                ROUND(len * COS(2 * PI() * angle)) AS y,
                ROUND(len * 0.3)::INTEGER + 1 AS trace
        FROM    (
                SELECT  RANDOM() AS angle,
                        8 * (1 - POWER(RANDOM(), 3)) AS len
                FROM    generate_series (1, 50) s
                ) q
        ),
        traces AS
        (
        SELECT  *,
                generate_series(1, trace) AS part
        FROM    centers
        ),
        parts AS
        (
        SELECT  CASE
                WHEN trace = part THEN
                        LEAST(len * 0.2, 2)::INTEGER
                ELSE
                        TRUNC(angle * 8 - 0.5)::INTEGER % 4 + 3
                END AS symbol,
                TRUNC(x + part * SIN(2 * PI() * angle)) AS x,
                TRUNC(y + part * COS(2 * PI() * angle)) AS y
        FROM    traces
        )
        SELECT  ARRAY_TO_STRING(
        ARRAY(
        SELECT  COALESCE(
                (
                SELECT  SUBSTR(E'.xX\\-/|', MIN(symbol) + 1, 1)
                FROM    parts
                WHERE   x = col - 14
                        AND y = row - 12
                ), ' ')
        FROM    generate_series(1, 25) col
        ), ''
        ) AS FIREWORKS
FROM    generate_series(1, 24) row
fireworks
X
| X
x| | X
x | | /
X\ \x | /x
X\ \ \ /
X \ \ x / /xx
\\ x |. ///
X– \ |
xxx– .
Xx– ..
xxx-/ –x
. \\ –X
. .
/ \
X/ | \ \x
/ |\ \\
// |x\ \ \x
x/ |\x \ X
X x\x X
X

MySQL postcard

SELECT  CONCAT(COALESCE(border, '|'), RPAD(COALESCE(postcard, ''), 61, COALESCE(filler, ' ')), COALESCE(border, '|')) AS postcard
FROM    (
        SELECT  id2 * 5 + id1 AS line
        FROM    (
                SELECT 1 AS id1 UNION ALL SELECT 2 UNION ALL SELECT 3 UNION ALL SELECT 4 UNION ALL SELECT 5
                ) q1
        CROSS JOIN
                (
                SELECT 0 AS id2 UNION ALL SELECT 1
                ) q2
        ) dummy
LEFT JOIN
        (
        SELECT  3 AS line, CONCAT(' Dear ', USER(), ',') AS postcard
        UNION ALL
        SELECT  4, ' I wish you good luck in the New Year.'
        UNION ALL
        SELECT  5, ' May all your plans be optimal, all your queries be answered'
        UNION ALL
        SELECT  6, ' and all your estimations be correct.'
        UNION ALL
        SELECT  8, '               Yours sincerely,'
        UNION ALL
        SELECT  9, '                                                    Quassnoi'
        ) ln
ON      ln.line = dummy.line
LEFT JOIN
        (
        SELECT 1 AS line, '.' AS border, '_' AS filler
        UNION ALL
        SELECT 10 AS line, '.' AS border, '_' AS filler
        ) borders
ON      borders.line = dummy.line
ORDER BY
        dummy.line
postcard
._____________________________________________________________.
| |
| Dear dbuser@localhost, |
| I wish you good luck in the New Year. |
| May all your plans be optimal, all your queries be answered |
| and all your estimations be correct. |
| |
| Yours sincerely, |
| Quassnoi |
._____________________________________________________________.
Happy New Year!
]]> http://explainextended.com/2009/12/31/happy-new-year/feed/ 5 What is entity-relationship model? http://explainextended.com/2009/10/18/what-is-entity-relationship-model/ http://explainextended.com/2009/10/18/what-is-entity-relationship-model/#comments Sun, 18 Oct 2009 19:00:46 +0000 Quassnoi http://explainextended.com/?p=2959 Image by Samuel Mann

Image by Samuel Mann

A relational database, as we all know from the previous article, stores relations between integers, strings and other simple types in a very plain way: it just enumerates all values related.

This model is extremely flexible, since other relations can be easily derived from existing ones using mathematical formulae, and the relational database takes care of that.

However, database should reflect the real world in some way to be really of use. Since the relational databases store relations between mathematical abstractions and not real world things, we should make some kind of a mapping of ones to the others.

This is what entity-relationship model is for.

An entity-relationship model, as one can easily guess from its name, models relationships between entities.

But since we know that databases do essentially the same, how does it differ from the database model?

  • An entity-relationship model states which data and relations between them should be stored
  • A database model states how these relations are stored

In other words, ER model is design and database model is one of the ways to implement it. ER model is said to be above the database model in the waterfall developement.

Note that in both cases I use the word stored above. The model says nothing of data and relations between them that may or should exist, only of those that should be stored. Every human being participates in thousands if not millions relationships, but an ER model should state which of them are to be stored, and a relational model should decide how to store them. Of course it would be nice to store them all (and police and insurance would be just happy), but the technology does not allow it yet. Don’t know whether it is good or bad.

A typical entity-relationship diagram in Peter Chen’s notation looks like this:

An entity-relationship diagram

What does it mean?

Entities

Square boxes mean entites. What do we describe in our database?

In our case, the model requires that we store information about clients, orders and
items ordered.

Square boxes are useless without additional information. Store some information? OK, we do. We just make a file to which a new line is added each time a new customer registers:

A customer has registered
A customer has registered
A customer has registered
…

, another file to which a new line is added when an order is made:

An order is made
An order is made
An order is made
…

and a third file to keep the list of items ordered.

An item is ordered
An item is ordered
…

Quite useless, right? However, there still is some information, as per request. We know that each row in the files describe a customer, an order or an item, and we can get the total number of customers, orders and items we’ve had so far.

This is better than nothing but still far from being usable.

Note that the diagram tells us nothing of how they should be stored. We could write them into a file, or on a paper or carve it on a stone.

Relationships

Diamonds mean relationships. How the entities are related to each other?

We see that we should store the relationships between orders and clients as well as those between items and orders.

More than that: tiny arrows between the diamonds and the boxes (that is, between the entities and relationships) show us which relationships should we store.

In our cases, we have a 1:(0-N) relationship between clients and orders and a 1:(0-N) relationship between orders and clients.

1:(0-N) is spelled as one to zero, one or many

That means that our database should make sure that any order is related to exactly one client.

Want to store an order? Be prepared to link it to the piece of information about the client who made it and make sure this information describes exactly one client. This is the database developer’s task.

However, a client can have an arbitrary number of orders: no orders at all, one order or many orders. The database should provide an ability to store the clients and orders this way.

Again, the entity-relationship model does not prescribe how should we store that data, as long as the storage method satisfies the conditions above.

We could store it in two files and relate them using the row numbers:

A customer has registered
A customer has registered
A customer has registered
…

An order is made by the customer described on line 1
An order is made by the customer described on line 2
An order is made by the customer described on line 2

, or we can just keep the information in a single text file:

A client has registered
Made an order
A client has registered
Made an order
Made an order
…

, or keep it in an XML file:

<root>
 <client>
   <order/>
 </client>
 <client>
   <order/>
   <order/>
 </client>
</root>

, or even bend down a finger each time a client makes an order. The latter one limits us to only 2 clients and 5 orders, but, you know, every system has its limitations.

Attributes

Ovals are attributes. What information is stored in the database?

Mere enumerating the clients is nice but serves no purpose. Much better if you know the names of the clients; the orders need to be assigned with unique numbers that help to identify them; and it would be great to record which good did which item contain (so not only the number of packages could be checked but their contents too).

This is in fact what the database is for: storing information. Not only the links between the entities but the descriptions of the entities too.

Something like this:

<root>
 <client name="John">
   <order number="1">
    <item name="apple"/>
   </order>
 </client>
 <client name="Jim">
   <order number="2">
    <item name="apple"/>
    <item name="banana"/>
   </order>
 </client>
</root>

That’s what makes database a database, not a computer-assisted finger-counting.

Entity-relationship and relational model

Everything above should be squeezed into relational model, which as we all know stores relations.

Since the time relational database appeared, they were mostly used to implement ER models. Multiple database manuals and guides describe the relational databases solely from that point of view. Various tools exist to automatically generate relational structure given a model.

However, ER model and a relational database are not the same. There is even no mapping to either side: same ER model can be implemented in different ways in relational database and vise versa, a relational structure can serve multiple ER models.

Due to the way the data are stored in a relational model, there is no reliable way to tell between attributes, entities and relationships by looking only at the relational model. These terms belong to the ER model. In a relational model, one thing can be implemented as an entity, relationship or an attribute.

In this article, I will give several examples.

Attribute or relationship?

Fictional character

Imagine a simple model as pictured in the diagram on the right.

The model requires that the database store fictional characters (as the entities). For each fictional character it should store their name, address, town and state (as the attributes).

There are no relations here: we store only one type of entities and their attributes.

As I already said earlier, the ER model specifies what should be stored and the database design (relational model in this case) decides how. One ot the benefits of the relational model is that is can construct the data representation on the fly, using SQL statements. That’s why there is more than one way to design a storage model, and as long as it’s possible to retrieve the data in expected form, every storage model is just as good.

One of the possible ways to implement this ER model in a relational database is to store the entities and their attributes in one table, like this:

FictionalCharacter
ID First Name Last Name Address Town State
1 Marty McFly 9303 Lyon Estates Hill Valley CA
2 Arnold 4040 Vineland Hillwood WA
3 Hank Hill 123 Rainey Street Arlen TX

This table serves two purposes: it both defines an entity and stores its attributes. The entities are defined by being listed in a table with their id (a value being used in the relations) defined as a PRIMARY KEY.

Not all fictional characters, though, have last names. Arnold of Hey, Arnold! does not.

There is one more way to implement this model: store the entities and attributes in two (or even more) separate relational tables. If an attribute is rarely set or rarely used, it can be offloaded into another table:

FictionalCharacter
ID First Name Address Town State
1 Marty 9303 Lyon Estates Hill Valley CA
2 Arnold 4040 Vineland Hillwood WA
3 Hank 123 Rainey Street Arlen TX
FictionalCharacterLastName
ID Last Name
1 McFly
3 Hill

The model remains the same: the last name is an attribute of an entity, possibly undefined; but the relational implementations differ. Nevertheless, we can easily transform one representation to another by issuing SQL queries.

This second one can be transformed to the first one using this query:

SELECT  c.*, ln.LastName
FROM    FictionalCharacter c
LEFT JOIN
        FictionalCharacterLastName ln
ON      ln.id = c.id

and vice versa:

SELECT  c.id, c.FirstName, ln.LastName, c.Address, c.Town, c.State
FROM    FictionalCharacter c
LEFT JOIN
        FictionalCharacterLastName ln
ON      ln.id = c.id

SELECT  c.id, c.LastName
FROM    FictionalCharacter c
WHERE   c.LastName IS NOT NULL

The first model (storing all attributes in one table) seems more simple. Why should one ever use the second solution which is more complex?

The reason is that the relational database, itself being an abstraction, should nevertheless be stored on a real world disk drive and served by a real world software. And when it comes to the real world, as we all know, several trade-offs should be made.

In the world of relational databases there is a convention to use a special value, NULL, to mark the data whose value is undefined, that is not known, not stored or not cared for. This value is not equal (and not even unequal) to any value (including itself), the results of most operators and functions over this value are also undefined.

However, it still should be stored in the database somehow. Most databases optimize it to occupy as few space as possible. Oracle, for instance, stored data in rows of dynamic size, the row itself and each column being prepended with their sizes. If the column count in a given row is less than it should be (i. e. there are fewer columns stored in a row than it is stated in the table definition), all other columns are considered to have the value of NULL. This makes storing NULLS in trailing columns to be free in terms of the disk drive space.

But sometimes NULL still occupies some space. Some systems store the data in rows of fixed length; some use no strict datatypes and therefore should store the database along with each value (rather than relying on the table definition); some (like Oracle) should store the NULL if the column is not trailing, i. e. there are some non-NULL values after it.

This makes the table bigger in size. And the queries that do not return this attribute are slower than they would be were it not for that column, since the data occupies more space and the engine needs to read more datapages.

On the other hand, the queries returning all columns, including that which is rarely used, are faster than they would be were this column stored in a separate table. Extra disk reads and CPU time are required to join these columns into one resulting relation and present it to the querying code.

The relations we get as a result of the queries are exactly same, since the relational databases separate logical presentation from the physical storage. But they do it does not mean they do it the best way. They do not know beforehand which query shall we do more often: the one that returns all attributes or the one that omits some of them.

The two queries, one returning the missing attribute and one lacking it, cannot be optimized together. Making one faster usually makes the other one slower. If the attribute is stored only for some, not for all entities, or the attribute is not queried for much often, it may be useful to store it in a separate table on the disk, so that the most used query is faster.

Both these storage methods implement one ER model. It’s only the way the attribute is stored is changed, not the way it is presented logically. We can even create a view or a stored procedure that would hide this storage method from the calling application completely.

In a relational database, the ER model defines the number and the possible values of the attributes but not the way they are stored.

The entity should be defined by a relation corresponding the PRIMARY KEY of the entity with all of its attributes. The database can store this relation in a relational table as is or generate it from the several tables on demand. As long as the database is able to do that, the model implementation is considered valid.

Storing something in another table and joining the tables to get the results usually means a relationship (in this case a one-to-one relationship). But in this model it is not a relationship that the separate table defines. It is just a more efficient way to store an attribute.

Attribute or entity?

Back to the model that describes fictional characters:

Fictional character

and its implementation:

FictionalCharacter
ID First Name Last Name Address Town State
1 Marty McFly 9303 Lyon Estates Hill Valley CA
2 Arnold 4040 Vineland Hillwood WA
3 Hank Hill 123 Rainey Street Arlen TX

Here we see an attribute, State, which contains a two-letter state code.

Now, what if we want to select all characters who live in states with population less than 10,000,000?

We don’t keep information about the population of the U.S. states in our model. We will need to add it to the model somehow.

We could possibly add an additional attribute, StatePopulation, that would show the population of the state the character lives in.

However, this is not the best design: whenever we need to reflect a change in state population in our model, we should update the records for all character who live in these states. In addition, we should make sure that all these records are consistent: all characters who live in same state should have same StatePopulation.

It would be wrong if two Hills lived in same Texas with different populations. StatePopulation is a property of a state, not that of a person who lives in that state.

But the entity-relationship model does not allow adding attributes to attributes. Instead, we should make State an entity, define a many-to-one relationship between the persons and the states and keep the Population as an attribute of a State.

Here’s how the new model would look:

Fictional character and state

This process is usually called normalization. However, normalization is relational concept, not the ER concept. Here is the definition of the third normal form (which was what we made here) from Wikipedia:

The third normal form (3NF) is a normal form used in database normalization. 3NF was originally defined by E.F. Codd in 1971. Codd’s definition states that a table is in 3NF if and only if both of the following conditions hold:

  • The relation R (table) is in second normal form (2NF)
  • Every non-prime attribute of R is non-transitively dependent (i.e. directly dependent) on every key of R.

As we can see, there are a lot of things in this definition that the ER model knows nothing about: relations, keys, tables. But the ER model has a similar concept: attributes cannot have the child attributes, entites can. If you want to define an attribute for an attribute, promote the latter to an entity and define the relationship. Transitive dependence is just another name for an attribute of an attribute.

Here is how this new ER model could be implemented in a relational database:

FictionalCharacter
ID First Name Last Name Address Town State
1 Marty McFly 9303 Lyon Estates Hill Valley CA
2 Arnold 4040 Vineland Hillwood WA
3 Hank Hill 123 Rainey Street Arlen TX
State
State Population
CA 36,756,666
TX 24,326,974
WA 6,549,224

To select all characters who live in states with population less than 10,000,000, we just issue the following query:

SELECT  FictionalCharacter.*
FROM    FictionalCharacter
JOIN    State
ON      State.State = FictionalCharacter.State
WHERE   State.Population < 10000000

Now lets see what have we changed here.

In the ER model:

  • State became an entity
  • One attribute, State was removed from FictionalCharacter
  • An additional relationship appeared: FictionalCharacter lives in State
  • An additional attribute, Population, was added to State

In the relational model:

  • One additional table, State, appeared.
  • One constraint, a FOREIGN KEY referencing State, was added to the FictionalCharacter

Nothing had changed in the original table except for the FOREIGN KEY constraint added to ensure the data integrity.

We see that promoting the State to an entity didn’t affect the original table. The field State remained. The only thing that changed for this attribute that it now refers a record in the new table, State, that both defines the list of states and holds the state populations.

This is because relational model (unlike ER model) does not really distinguish between the attributes and entities.

Relational model stores relations between the simple types (like integers and strings). A real world relational database stores relations between the integers and strings that describe some real world data.

A U.S. state does not become less of an entity, be it listed as an entity or as an attribute in the ER model. If it’s enough for us to know only the state code (which is used in the relational model to represent the state in the relations stored in the database), we can store it in an unconstrained field; if it is not, we should create a separate table for it and constrain the value of the field using a FOREIGN KEY.

In this very case it’s easy to tell that there is a relation by looking into the table structure. There is a FOREIGN KEY that gives us a hint that the State is actually a field defining a one-to-many relationship, not a mere attribute.

But’s let’s consider another model:

Goods and Bonuses

This model described goods and price-based bonuses. The price ranges should be contiguous and the last range should have no upper bound.

Here’s how it would look in a relational database:

Goods
ID Name Price
1 Wormy apple 0.09
2 Bangkok durian 9.99
3 Densuke watermelon 999.99
4 White truffle 99999.99
PriceRange
Price Bonus
0.01 1%
1.00 3%
100.00 10%
10000.00 30%

Does the PriceRange table really define the price range entities?

Sure it does. It contains all required information: the StartPrice (as a field), the EndPrice (as the same field in the next record in PriceOrder) and the Bonus (as a field in the record that defines the StartPrice). It is possible to write a query that would return a classical relation with all three attributes in one record:

SELECT  Price AS StartPrice,
        (
        SELECT  MIN(Price)
        FROM    PriceRange NextRange
        WHERE   NextRange.Price > CurrentRange.Price
        ) AS EndPrice,
        Bonus
FROM    PriceRange CurrentRange

Due to the way the entity is defined, it is impossible to put a FOREIGN KEY constraint to a price range. The entity price range is not even being defined by a relation stored in a database: instead, a more simple relation is stored, which corresponds the value of the attribute (Bonus) with a lower bound of the price range.

To build the relation which would correspond each good with its price range and the bonus, we should issue the following query:

SELECT  *
FROM    Goods
JOIN    PriceRange
ON      PriceRange.Price =
        (
        SELECT  MAX(Price)
        FROM    PriceRange
        WHERE   PriceRange.Price <= Goods.Price
        )

We could probably use a more formalistic approach to the problem: store PriceRange in a table with a surrogate PRIMARY KEY, the lower and upper bounds as the attributes; and store a FOREIGN KEY reference to it in Goods.

This design would be more familiar but it could easily lead to data inconsistency. What if the Price of a good is not within the PriceRange the good references? What is the price ranges are not contiguous? What if the price ranges values are updated and the referencing columns are not?

The root of this problem it that the mechanisms to ensure referential integrity work on declared relationships, i. e. those just listed in a table, while what we deal with here is an inferred relationship.

The relationship between a good and a price range is not defined by a mere declaration of the fact that the relationship exists, i. e. it is not taken from the outside world. Instead, whether the relationship exists or not is defined by the values of the attributes of a good and a price range. It is generated from the other information stored in the database. And the relational databases, as I already said above, can transform the relations they store in the tables to other relations, using the relational algebra, mathematical operators and other means.

This relational design implements the ER model more consistently and efficiently, and this implementation is less prone to errors. However, using this design, we cannot use automated tools to map the relational model back to the ER model anymore. We don’t define the PriceRange and both of its attributes using a single stored relation referenceable with a FOREIGN KEY. From the relational model’s point of view, the Price is just an attribute of a Good which has no reference to the PriceRange. Nevertheless, the PriceRange is an entity with its attributes (which can only be represented by self-joining the table), and Price is used to reference it.

The fact that this design is more consistent, however, does not mean that the original design should never be used. Database normalization helps to cope with the possible inconsistencies in the model by inferring relations rather than storing them, but this can require additional resources (memory, CPU etc) and therefore be less efficient. There are numerous situations which require to store relations in a way that allows possible logical inconsistencies but is faster to query. Keeping the model consistent now becomes a developer’s task but it is the price one has to pay for speed. This process is called denormalization.

I will not discuss benefits and drawbacks of normalization and denormalization in this article. What I wanted to say is that usually there are many ways to implement an ER model in a relational database and not all these methods can be easily mapped back to the ER model which clearly distinguishes between entities, attributes and relationships.

A relational model uses same datatypes to represent entities and to define attributes. Therefore, it is best thought of as a way to store relations between the entities of the real world (represented by the numbers and strings), some of which are not of interest enough to be the entities of the model.

Whether the values the table relates represent entities or attributes is not a concern of a relational model. It can store and return both with equal efficiency.

Relationship or entity?

Consider this model:

Couples (one-to-one)

It describes fictional persons who are married to each other.

This can be implemented in a relational database as this:

FictionalPerson
ID Gender First Name Last Name
1 M Desmond Jones
2 F Mary Jones
Marriage
ID Husband Wife
1 1 2

Here we see that Desmond Jones is married to Mary Jones. Marriage of course is relationship (and is called so even outside the database world).

That’s fine. Now we want to add Scarlett O’Hara into the model. She was married first to Ashley Wilkes, then to Rhett Butler.

The model as it is now does not let us do it, since a marriage is a one-to-one relation (at least in the Christian tradition). We should change this relation to a many-to-many (which changes its wording from is married to to has ever been married to).

Here’s how the new model would look like:

Couples (many-to-many)

FictionalPerson
ID Gender First Name Last Name
1 M Desmond Jones
2 F Mary Jones
3 M Ashley Wilkes
4 M Rhett Butler
5 F Scarlett O’Hara
Marriage
ID Husband Wife
1 1 2
2 3 5
3 4 5

The ER model changed the relationship from one-to-one to many-to-many.

To reflect this, the relational model dropped the separate UNIQUE constraints on Husband and Wife and added a composite UNIQUE constraint on (Husband, Wife).

Now, let’s the add two more persons to our model: Ransie and Ariela Bilbro from O. Henry’s The Whirligig of Life. They married each other twice.

This time even changing the relationship type in the ER model won’t help. There is no such thing as a double relationship. To reflect the double marriages, the relationship should be promoted to a first-class entity with one-to-one relation to either part (which now means been in this marriage).

The model would look like this now:

CouplesMarriages

We see that the ER model changed significantly. How did the relational model change?

Since the husband and wife do not uniquely define the marriage anymore, the UNIQUE constraint should be dropped from the Marriage, but otherwise the tables remains the same:

FictionalPerson
ID Gender First Name Last Name
1 M Desmond Jones
2 F Mary Jones
3 M Ashley Wilkes
4 M Rhett Butler
5 F Scarlett O’Hara
6 M Ransie Bilbro
7 F Ariela Bilbro
Marriage
ID Husband Wife
1 1 2
2 3 5
3 4 5
4 6 7
5 6 7

This happened because the marriage in fact is an entity, with its own attributes like date and place of the wedding, the number of guests invited, awful toasts made, tea sets presented etc. But the ER model was not initially interested in all this and the marriage was declared as a relationship, not an entity.

In ER model, a relationship between two entities can either exist or not exist, one or the other. There cannot be such thing as a double relationship or expired relationship. To trace such things one should need to promote a relationship to an entity and assign the attributes to it.

However, the relation model can handle this easily. The way the relationship is build in our implementation, there is no clear disctinction between a marriage as a relationship and a marriage as an entity. By putting and removing UNIQUE constraints, Marriage can be easily changed to represent a one-to-one relationship, a many-to-many relationship or even an entity and two one-to-one relationships at once, the relational struсture still remaining the same.

Summary

The entity-relationship model defines what should be stored in a database: about what, which information, how related. This should be described in terms of entities, relationships and attributes.

A relational model describes how to implement the requirements of the ER model: which information should be stored in which relational tables.

The relational model is very flexible and can construct relations on the fly. The relations can be stored in one way and represented in another way. This helps in handling more complex models that require the relationships to be inferred rather than stored. The database can thus be made more maintainable and less prone to logical inconsistencies.

Since the relational model just defines relations between simple types like integers and strings, things like entities, attributes and relationships are not a concern of a relational model. It only implements the requirements established by the entity-relationship model.

Usually it does it using simple and formal mechanisms: a PRIMARY KEY-preserved table to define entities and attributes, and a separate table with FOREIGN KEY references to define the relationships.

However, it can change the storage methods: move the attributes and relationship between the tables, define the entities so that the attributes should be calculated, etc. This can be done both to improve efficiency and to get rid of inconsistencies.

Since relational model separates storage from presentation by using a built-in relation transformation language, SQL, any storage method is valid as long as it is able to construct the entities, attributes and relationships in their canonical form.

That’s why it makes no sense to speak of the entities, relationships and attributes in respect to the relational model. These things are defined by the entity-relationship model.

]]> http://explainextended.com/2009/10/18/what-is-entity-relationship-model/feed/ 12