EXPLAIN EXTENDED » SQL Server http://explainextended.com How to create fast database queries Mon, 02 Jan 2012 00:31:26 +0000 en hourly 1 http://wordpress.org/?v=3.3.1 What’s UNPIVOT good for? http://explainextended.com/2011/06/30/whats-unpivot-good-for/ http://explainextended.com/2011/06/30/whats-unpivot-good-for/#comments Thu, 30 Jun 2011 19:00:56 +0000 Quassnoi http://explainextended.com/?p=5373 Answering questions asked on the site.

Karen asks:

… I’ve always thought PIVOT and UNPIVOT are signs of a poorly designed database. I mean, is there a legitimate use for them if your model is OK?

I’ve made an actual use for them in a project I’ve been working on for the last several months (which is partly why there were no updates for so long!)

Part of the project is a task management system where each task has several persons related to it. There can be the creator of the task, the person the task is assigned to, the actual author of the task (on behalf of whom the task is created), and the task can be possible completed by a person or deleted by a person. A total of 5 fields related to persons.

Now, we need to take all tasks within a certain time range and list all people involved in them.

Let’s create a sample table and see how would we do that.

Table creation details

There are 5 persons fields, timestamp and stuffing. The timestamp field is indexed.

Now, let’s find all people involved in the tasks between 2011-06-30 and 2011-06-30 04:00:00. To do this, we could just use 5 queries (each selecting one of the persons) and UNION them:

SELECT  SUM(CAST(id AS BIGINT)), COUNT(*)
FROM    (
        SELECT  createdBy AS id
        FROM    [20110630_unpivot].Task
        WHERE   ts >= '2011-06-30'
                AND ts < '2011-06-30 04:00:00'
        UNION
        SELECT  assignedTo AS id
        FROM    [20110630_unpivot].Task
        WHERE   ts >= '2011-06-30'
                AND ts < '2011-06-30 04:00:00'
        UNION
        SELECT  onBehalfOf AS id
        FROM    [20110630_unpivot].Task
        WHERE   ts >= '2011-06-30'
                AND ts < '2011-06-30 04:00:00'
        UNION
        SELECT  completedBy AS id
        FROM    [20110630_unpivot].Task
        WHERE   ts >= '2011-06-30'
                AND ts < '2011-06-30 04:00:00'
                AND completedBy IS NOT NULL
        UNION
        SELECT  deletedBy AS id
        FROM    [20110630_unpivot].Task
        WHERE   ts >= '2011-06-30'
                AND ts < '2011-06-30 04:00:00'
                AND deletedBy IS NOT NULL
        ) q
2573160101 51331
1 row fetched in 0.0009s (0.4251s) 
[Microsoft][SQL Server Native Client 10.0][SQL Server]Table 'Task'. Scan count 15, logical reads 220765, physical reads 0, read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.
[Microsoft][SQL Server Native Client 10.0][SQL Server]Table 'Worktable'. Scan count 0, logical reads 0, physical reads 0, read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.
[Microsoft][SQL Server Native Client 10.0][SQL Server]Table 'Worktable'. Scan count 0, logical reads 0, physical reads 0, read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.
[Microsoft][SQL Server Native Client 10.0][SQL Server]
 SQL Server Execution Times:
   CPU time = 484 ms,  elapsed time = 410 ms.

  |--Compute Scalar(DEFINE:([Expr1016]=CASE WHEN [globalagg1019]=(0) THEN NULL ELSE [globalagg1021] END, [Expr1017]=CONVERT_IMPLICIT(int,[globalagg1023],0)))
       |--Stream Aggregate(DEFINE:([globalagg1019]=SUM([partialagg1018]), [globalagg1021]=SUM([partialagg1020]), [globalagg1023]=SUM([partialagg1022])))
            |--Parallelism(Gather Streams)
                 |--Compute Scalar(DEFINE:([partialagg1022]=[partialagg1018]))
                      |--Stream Aggregate(DEFINE:([partialagg1018]=Count(*), [partialagg1020]=SUM(CONVERT(bigint,[Union1015],0))))
                           |--Hash Match(Aggregate, HASH:([Union1015]), RESIDUAL:([Union1015] = [Union1015]))
                                |--Concatenation
                                     |--Parallelism(Repartition Streams, Hash Partitioning, PARTITION COLUMNS:([ee].[20110630_unpivot].[Task].[createdBy]))
                                     |    |--Nested Loops(Inner Join, OUTER REFERENCES:([ee].[20110630_unpivot].[Task].[id], [Expr1025]) OPTIMIZED WITH UNORDERED PREFETCH)
                                     |         |--Index Seek(OBJECT:([ee].[20110630_unpivot].[Task].[IX_Task_Ts]), SEEK:([ee].[20110630_unpivot].[Task].[ts] >= '2011-06-30 00:00:00.000' AND [ee].[20110630_unpivot].[Task].[ts] < '2011-06-30 04:00:00.000') ORDERED FORWARD)
                                     |         |--Clustered Index Seek(OBJECT:([ee].[20110630_unpivot].[Task].[PK__Task__3213E83F0B3E4B07]), SEEK:([ee].[20110630_unpivot].[Task].[id]=[ee].[20110630_unpivot].[Task].[id]) LOOKUP ORDERED FORWARD)
                                     |--Parallelism(Repartition Streams, Hash Partitioning, PARTITION COLUMNS:([ee].[20110630_unpivot].[Task].[assignedTo]))
                                     |    |--Nested Loops(Inner Join, OUTER REFERENCES:([ee].[20110630_unpivot].[Task].[id], [Expr1026]) OPTIMIZED WITH UNORDERED PREFETCH)
                                     |         |--Index Seek(OBJECT:([ee].[20110630_unpivot].[Task].[IX_Task_Ts]), SEEK:([ee].[20110630_unpivot].[Task].[ts] >= '2011-06-30 00:00:00.000' AND [ee].[20110630_unpivot].[Task].[ts] < '2011-06-30 04:00:00.000') ORDERED FORWARD)
                                     |         |--Clustered Index Seek(OBJECT:([ee].[20110630_unpivot].[Task].[PK__Task__3213E83F0B3E4B07]), SEEK:([ee].[20110630_unpivot].[Task].[id]=[ee].[20110630_unpivot].[Task].[id]) LOOKUP ORDERED FORWARD)
                                     |--Parallelism(Repartition Streams, Hash Partitioning, PARTITION COLUMNS:([ee].[20110630_unpivot].[Task].[onBehalfOf]))
                                     |    |--Nested Loops(Inner Join, OUTER REFERENCES:([ee].[20110630_unpivot].[Task].[id], [Expr1027]) OPTIMIZED WITH UNORDERED PREFETCH)
                                     |         |--Index Seek(OBJECT:([ee].[20110630_unpivot].[Task].[IX_Task_Ts]), SEEK:([ee].[20110630_unpivot].[Task].[ts] >= '2011-06-30 00:00:00.000' AND [ee].[20110630_unpivot].[Task].[ts] < '2011-06-30 04:00:00.000') ORDERED FORWARD)
                                     |         |--Clustered Index Seek(OBJECT:([ee].[20110630_unpivot].[Task].[PK__Task__3213E83F0B3E4B07]), SEEK:([ee].[20110630_unpivot].[Task].[id]=[ee].[20110630_unpivot].[Task].[id]) LOOKUP ORDERED FORWARD)
                                     |--Parallelism(Repartition Streams, Hash Partitioning, PARTITION COLUMNS:([ee].[20110630_unpivot].[Task].[completedBy]))
                                     |    |--Nested Loops(Inner Join, OUTER REFERENCES:([ee].[20110630_unpivot].[Task].[id], [Expr1028]) OPTIMIZED WITH UNORDERED PREFETCH)
                                     |         |--Index Seek(OBJECT:([ee].[20110630_unpivot].[Task].[IX_Task_Ts]), SEEK:([ee].[20110630_unpivot].[Task].[ts] >= '2011-06-30 00:00:00.000' AND [ee].[20110630_unpivot].[Task].[ts] < '2011-06-30 04:00:00.000') ORDERED FORWARD)
                                     |         |--Clustered Index Seek(OBJECT:([ee].[20110630_unpivot].[Task].[PK__Task__3213E83F0B3E4B07]), SEEK:([ee].[20110630_unpivot].[Task].[id]=[ee].[20110630_unpivot].[Task].[id]),  WHERE:([ee].[20110630_unpivot].[Task].[completedBy] IS NOT NULL) LOOKUP ORDERED FORWARD)
                                     |--Parallelism(Repartition Streams, Hash Partitioning, PARTITION COLUMNS:([ee].[20110630_unpivot].[Task].[deletedBy]))
                                          |--Nested Loops(Inner Join, OUTER REFERENCES:([ee].[20110630_unpivot].[Task].[id], [Expr1029]) OPTIMIZED WITH UNORDERED PREFETCH)
                                               |--Index Seek(OBJECT:([ee].[20110630_unpivot].[Task].[IX_Task_Ts]), SEEK:([ee].[20110630_unpivot].[Task].[ts] >= '2011-06-30 00:00:00.000' AND [ee].[20110630_unpivot].[Task].[ts] < '2011-06-30 04:00:00.000') ORDERED FORWARD)
                                               |--Clustered Index Seek(OBJECT:([ee].[20110630_unpivot].[Task].[PK__Task__3213E83F0B3E4B07]), SEEK:([ee].[20110630_unpivot].[Task].[id]=[ee].[20110630_unpivot].[Task].[id]),  WHERE:([ee].[20110630_unpivot].[Task].[deletedBy] IS NOT NULL) LOOKUP ORDERED FORWARD)

Now we have 51331 persons returned in 410 ms, which required 220765 logical reads.

As we can see in the plan, the table is scanned 5 times (once for each query). This is not very efficient of course. It would be much better if each matching record in the table would be only visited once.

This is where UNPIVOT comes into play.

As its name suggests, it does the reverse of PIVOT, that is moves data from multiple columns to multiple columns. And this is exactly what we need in our case.

Let’s try it:

SELECT  SUM(CAST(personId AS BIGINT)), COUNT(*)
FROM    (
        SELECT  DISTINCT personId
        FROM    [20110630_unpivot].Task
        UNPIVOT
                (
                personId FOR personType IN
                (createdBy, assignedTo, onBehalfOf, completedBy, deletedBy)
                ) p
        WHERE   ts >= '2011-06-30'
                AND ts < '2011-06-30 04:00:00'
        ) q
2573160101 51331
1 row fetched in 0.0002s (0.1802s) 
[Microsoft][SQL Server Native Client 10.0][SQL Server]Table 'Worktable'. Scan count 0, logical reads 0, physical reads 0, read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.
[Microsoft][SQL Server Native Client 10.0][SQL Server]Table 'Task'. Scan count 3, logical reads 44153, physical reads 0, read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.
[Microsoft][SQL Server Native Client 10.0][SQL Server]Table 'Worktable'. Scan count 0, logical reads 0, physical reads 0, read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.
[Microsoft][SQL Server Native Client 10.0][SQL Server]
 SQL Server Execution Times:
   CPU time = 165 ms,  elapsed time = 180 ms.

  |--Compute Scalar(DEFINE:([Expr1013]=CASE WHEN [globalagg1016]=(0) THEN NULL ELSE [globalagg1018] END, [Expr1014]=CONVERT_IMPLICIT(int,[globalagg1020],0)))
       |--Stream Aggregate(DEFINE:([globalagg1016]=SUM([partialagg1015]), [globalagg1018]=SUM([partialagg1017]), [globalagg1020]=SUM([partialagg1019])))
            |--Parallelism(Gather Streams)
                 |--Compute Scalar(DEFINE:([partialagg1019]=[partialagg1015]))
                      |--Stream Aggregate(DEFINE:([partialagg1015]=Count(*), [partialagg1017]=SUM(CONVERT(bigint,[Expr1011],0))))
                           |--Sort(DISTINCT ORDER BY:([Expr1011] ASC))
                                |--Parallelism(Repartition Streams, Hash Partitioning, PARTITION COLUMNS:([Expr1011]))
                                     |--Hash Match(Partial Aggregate, HASH:([Expr1011]), RESIDUAL:([Expr1011] = [Expr1011]))
                                          |--Filter(WHERE:([Expr1011] IS NOT NULL))
                                               |--Nested Loops(Left Outer Join, OUTER REFERENCES:([ee].[20110630_unpivot].[Task].[createdBy], [ee].[20110630_unpivot].[Task].[assignedTo], [ee].[20110630_unpivot].[Task].[onBehalfOf], [ee].[20110630_unpivot].[Task].[completedBy], [ee].[20110630_unpivot].[Task].[deletedBy]))
                                                    |--Nested Loops(Inner Join, OUTER REFERENCES:([ee].[20110630_unpivot].[Task].[id], [Expr1022]) OPTIMIZED WITH UNORDERED PREFETCH)
                                                    |    |--Index Seek(OBJECT:([ee].[20110630_unpivot].[Task].[IX_Task_Ts]), SEEK:([ee].[20110630_unpivot].[Task].[ts] >= '2011-06-30 00:00:00.000' AND [ee].[20110630_unpivot].[Task].[ts] < '2011-06-30 04:00:00.000') ORDERED FORWARD)
                                                    |    |--Clustered Index Seek(OBJECT:([ee].[20110630_unpivot].[Task].[PK__Task__3213E83F14C7B541]), SEEK:([ee].[20110630_unpivot].[Task].[id]=[ee].[20110630_unpivot].[Task].[id]) LOOKUP ORDERED FORWARD)
                                                    |--Constant Scan(VALUES:(([ee].[20110630_unpivot].[Task].[createdBy]),([ee].[20110630_unpivot].[Task].[assignedTo]),([ee].[20110630_unpivot].[Task].[onBehalfOf]),([ee].[20110630_unpivot].[Task].[completedBy]),([ee].[20110630_unpivot].[Task].[deletedBy])))

As we can see, this is more than twice as efficient as a UNION and only takes 44153 page reads to complete.

How does it work internally?

We see in the plan that there is a nested loop between a result of a clustered index seek and something called Constant Scan. The constant scan returns 5 values in each loop and those are the fields listed in the UNPIVOT clause. It just takes each record and outputs fields found there, without rereading the record. This is actually what we wanted.

This behavior can be made more clear if we rewrite the query a little:

SELECT  SUM(CAST(personId AS BIGINT)), COUNT(*)
FROM    (
        SELECT  DISTINCT personId
        FROM    [20110630_unpivot].Task
        CROSS APPLY
                (
                SELECT  createdBy AS personId
                UNION ALL
                SELECT  assignedTo AS personId
                UNION ALL
                SELECT  onBehalfOf AS personId
                UNION ALL
                SELECT  completedBy AS personId
                UNION ALL
                SELECT  deletedBy AS personId
                ) p
        WHERE   ts >= '2011-06-30'
                AND ts < '2011-06-30 04:00:00'
                AND personId IS NOT NULL
        ) q
2573160101 51331
1 row fetched in 0.0002s (0.1766s) 
[Microsoft][SQL Server Native Client 10.0][SQL Server]Table 'Worktable'. Scan count 0, logical reads 0, physical reads 0, read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.
[Microsoft][SQL Server Native Client 10.0][SQL Server]Table 'Task'. Scan count 3, logical reads 44153, physical reads 0, read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.
[Microsoft][SQL Server Native Client 10.0][SQL Server]Table 'Worktable'. Scan count 0, logical reads 0, physical reads 0, read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.
[Microsoft][SQL Server Native Client 10.0][SQL Server]
 SQL Server Execution Times:
   CPU time = 149 ms,  elapsed time = 176 ms.

  |--Compute Scalar(DEFINE:([Expr1004]=CASE WHEN [globalagg1007]=(0) THEN NULL ELSE [globalagg1009] END, [Expr1005]=CONVERT_IMPLICIT(int,[globalagg1011],0)))
       |--Stream Aggregate(DEFINE:([globalagg1007]=SUM([partialagg1006]), [globalagg1009]=SUM([partialagg1008]), [globalagg1011]=SUM([partialagg1010])))
            |--Parallelism(Gather Streams)
                 |--Compute Scalar(DEFINE:([partialagg1010]=[partialagg1006]))
                      |--Stream Aggregate(DEFINE:([partialagg1006]=Count(*), [partialagg1008]=SUM(CONVERT(bigint,[Union1003],0))))
                           |--Sort(DISTINCT ORDER BY:([Union1003] ASC))
                                |--Parallelism(Repartition Streams, Hash Partitioning, PARTITION COLUMNS:([Union1003]))
                                     |--Hash Match(Partial Aggregate, HASH:([Union1003]), RESIDUAL:([Union1003] = [Union1003]))
                                          |--Filter(WHERE:([Union1003] IS NOT NULL))
                                               |--Nested Loops(Inner Join, OUTER REFERENCES:([ee].[20110630_unpivot].[Task].[createdBy], [ee].[20110630_unpivot].[Task].[assignedTo], [ee].[20110630_unpivot].[Task].[onBehalfOf], [ee].[20110630_unpivot].[Task].[completedBy], [ee].[20110630_unpivot].[Task].[deletedBy]))
                                                    |--Nested Loops(Inner Join, OUTER REFERENCES:([ee].[20110630_unpivot].[Task].[id], [Expr1013]) OPTIMIZED WITH UNORDERED PREFETCH)
                                                    |    |--Index Seek(OBJECT:([ee].[20110630_unpivot].[Task].[IX_Task_Ts]), SEEK:([ee].[20110630_unpivot].[Task].[ts] >= '2011-06-30 00:00:00.000' AND [ee].[20110630_unpivot].[Task].[ts] < '2011-06-30 04:00:00.000') ORDERED FORWARD)
                                                    |    |--Clustered Index Seek(OBJECT:([ee].[20110630_unpivot].[Task].[PK__Task__3213E83F14C7B541]), SEEK:([ee].[20110630_unpivot].[Task].[id]=[ee].[20110630_unpivot].[Task].[id]) LOOKUP ORDERED FORWARD)
                                                    |--Constant Scan(VALUES:(([ee].[20110630_unpivot].[Task].[createdBy]),([ee].[20110630_unpivot].[Task].[assignedTo]),([ee].[20110630_unpivot].[Task].[onBehalfOf]),([ee].[20110630_unpivot].[Task].[completedBy]),([ee].[20110630_unpivot].[Task].[deletedBy])))

Here, we just take each record and explode it into 5 records using CROSS APPLY.

This yields exactly same plan, exactly same I/O and exactly same output. In fact, that’s exactly what the UNPIVOT query does.

Hope that helps.

I’m always glad to answer the questions regarding database queries.

Ask me a question

]]> http://explainextended.com/2011/06/30/whats-unpivot-good-for/feed/ 5 GROUP_CONCAT in SQL Server http://explainextended.com/2010/06/21/group_concat-in-sql-server/ http://explainextended.com/2010/06/21/group_concat-in-sql-server/#comments Mon, 21 Jun 2010 19:00:49 +0000 Quassnoi http://explainextended.com/?p=4818 I’m finally back from my vacation. Tunisia’s great: dates, Carthage, sea and stuff.

Now, to the questions.

Mahen asks:

Create a table called Group:

Group
id prodname
1 X
1 Y
1 Z
2 A
2 B
2 C

The resultset should look like this:

id prodname
1 X,Y,Z
2 A,B,C

Can you please help me to solve the above problem using a recursive CTE?

This is out good old friend, GROUP_CONCAT. It’s an aggregate function that returns all strings within a group, concatenated. It’s somewhat different from the other aggregate functions, because, first, dealing with the concatenated string can be quite a tedious task for the groups with lots of records (large strings tend to overflow), and, second, the result depends on the order of the arguments (which is normally not the case for the aggregate functions). It’s not a part of a standard SQL and as for now is implemented only by MySQL with some extra vendor-specific keywords (like ORDER BY within the argument list).

This functionality, however, is often asked for and I have written some articles about implementing this in PostgreSQL and Oracle.

Now, let’s see how to do it in SQL Server.

Usually, SQL Server‘s FOR XML clause is exploited to concatenate the strings. To do this, we obtain a list of group identifiers and for each group, retrieve all it’s product names with a subquery appended with FOR XML PATH(''). This makes a single XML column out of the recordset:

WITH    q (id, prodname) AS
        (
        SELECT  1, 'X'
        UNION ALL
        SELECT  1, 'Y'
        UNION ALL
        SELECT  1, 'Z'
        UNION ALL
        SELECT  2, 'A'
        UNION ALL
        SELECT  2, 'B'
        UNION ALL
        SELECT  2, 'C'
        )
SELECT  *
FROM    (
        SELECT  DISTINCT id
        FROM    q
        ) qo
CROSS APPLY
        (
        SELECT  CASE ROW_NUMBER() OVER(ORDER BY prodname) WHEN 1 THEN '' ELSE ', ' END + qi.prodname
        FROM    q qi
        WHERE   qi.id = qo.id
        ORDER BY
                prodname
        FOR XML PATH ('')
        ) qi(r)
id r
1 X, Y, Z
2 A, B, C

This solution works, but converting to and from XML is not the best way to deal with the strings: things like ampersands, angle brackets, line feeds etc. get mangled and require some additional effort to cope with.

However, this functionality can really be implemented using a recursive CTE.

To do this, we need to do the following:

  1. Assign a group-wise row number and a group-wise count to each record (in required order)
  2. Select the first record (that with the row number 1) from each group in the anchor part of the CTE
  3. Recursively append the next record to the previous record. The next record can be obtained by joining on rn = rn + 1
  4. Finally, select the only last record from each group (whose row number is equal to the group-wise count). It will contain the accumulated string.

Here’s how we do it:

WITH    q (id, prodname) AS
        (
        SELECT  1, 'X'
        UNION ALL
        SELECT  1, 'Y'
        UNION ALL
        SELECT  1, 'Z'
        UNION ALL
        SELECT  2, 'A'
        UNION ALL
        SELECT  2, 'B'
        UNION ALL
        SELECT  2, 'C'
        ),
        qs(id, prodname, rn, cnt) AS
        (
        SELECT  id, prodname,
                ROW_NUMBER() OVER (PARTITION BY id ORDER BY prodname),
                COUNT(*) OVER (PARTITION BY id)
        FROM    q
        ),
        t (id, prodname, gc, rn, cnt) AS
        (
        SELECT  id, prodname,
                CAST(prodname AS NVARCHAR(MAX)), rn, cnt
        FROM    qs
        WHERE   rn = 1
        UNION ALL
        SELECT  qs.id, qs.prodname,
                CAST(t.gc + ', ' + qs.prodname AS NVARCHAR(MAX)),
                qs.rn, qs.cnt
        FROM    t
        JOIN    qs
        ON      qs.id = t.id
                AND qs.rn = t.rn + 1
        )
SELECT  id, gc
FROM    t
WHERE   rn = cnt
OPTION (MAXRECURSION 0)
id gc
2 A, B, C
1 X, Y, Z

As we can see, this only deals with native NVARCHAR and is free from XML conversions.

Hope that helps.

I’m always glad to answer the questions regarding database queries.

Ask me a question

]]> http://explainextended.com/2010/06/21/group_concat-in-sql-server/feed/ 0 SQL Server: deleting with self-referential FOREIGN KEY http://explainextended.com/2010/03/03/sql-server-deleting-with-self-referential-foreign-key/ http://explainextended.com/2010/03/03/sql-server-deleting-with-self-referential-foreign-key/#comments Wed, 03 Mar 2010 20:00:45 +0000 Quassnoi http://explainextended.com/?p=4539 From Stack Overflow:

I have an SQL Server table defined as below:

TestComposite
id (PK) siteUrl (PK) name parentId
1 site1 Item1 NULL
2 site1 Item2 NULL
3 site1 Folder1 NULL
4 site1 Folder1.Item1 3
5 site1 Folder1.Item2 3
6 site1 Folder1.Folder1 3
7 site1 Folder1.Folder1.Item1 6

Items and folders are stored inside the same table

If an item is inside a folder, the parentID column is the id of the folder.

I would like to be able to DELETE CASCADE items/folders when I delete a folder.

I tried to define a constraint similar to:

ALTER TABLE [TestComposite]
ADD CONSTRAINT fk_parentid
FOREIGN KEY (ParentID, SiteUrl)
REFERENCES [TestComposite] (ID, SiteUrl) ON DELETE CASCADE

, but it gives me this error:

Introducing FOREIGN KEY constraint 'fk_parentid' on table 'TestComposite' may cause cycles or multiple cascade paths. Specify ON DELETE NO ACTION or ON UPDATE NO ACTION, or modify other FOREIGN KEY constraints.

SQL Server does support chained CASCADE updates, but does not allow one table to participate more that once in a chain (i. e. does not allow loops).

SQL Server, unlike most other engines, optimizes cascading DML operations to be set-based which requires building a cycle-free DML order (which you can observe in the execution plan). With the loops, that would not be possible.

However, it is possible to define such a constraint without cascading operations, and with a little effort it is possible to delete a whole tree branch at once.

Let’s create a sample table:

Table creation details

This table contains 50,000 records forming a hierarchy.

If we try to delete an entry that has some children, we’ll fail:

DELETE
FROM    [20100303_cascade].TestComposite
WHERE   id = 42
        AND siteUrl = 'site1'

Msg 547, Level 16, State 0, Line 1
The DELETE statement conflicted with the SAME TABLE REFERENCE constraint "fk_TestComposite_self". The conflict occurred in database "test", table "20100303_cascade.TestComposite".
The statement has been terminated.

To delete an item and all of its children, we should build a hierarchical query to retrieve the whole branch, and delete the branch all at once.

To do it, we just semi-join the table to the results of the recursive CTE:

WITH    q AS
        (
        SELECT  id, siteUrl
        FROM    [20100303_cascade].TestComposite
        WHERE   id = 42
                AND siteUrl = 'site1'
        UNION ALL
        SELECT  tc.id, tc.siteUrl
        FROM    q
        JOIN    [20100303_cascade].TestComposite tc
        ON      tc.parentID = q.id
                AND tc.siteUrl = q.siteUrl
        )
DELETE
FROM    [20100303_cascade].TestComposite
OUTPUT  DELETED.*
WHERE   EXISTS
        (
        SELECT  id, siteUrl
        INTERSECT
        SELECT  id, siteUrl
        FROM    q
        )

View query results

We see that the whole branch of 31 records was deleted all at once, without violating the FOREIGN KEY.

Unlike some other systems, we don’t have to worry about the order the records are deleted, since all SQL Server referential constraints are deferred till the end of the query.

]]> http://explainextended.com/2010/03/03/sql-server-deleting-with-self-referential-foreign-key/feed/ 0 Efficient circle distance testing http://explainextended.com/2010/02/26/efficient-circle-distance-testing/ http://explainextended.com/2010/02/26/efficient-circle-distance-testing/#comments Fri, 26 Feb 2010 20:00:41 +0000 Quassnoi http://explainextended.com/?p=4449 Answering questions asked on the site.

eptil asks:

I am using SQL Server 2008, but not the spatial features.

I have a table with few entries, only 40,000. There is an id INT PRIMARY KEY column and two columns storing a 2d coordinate, both decimals.

I would like to find all the records that do not have other records within a given radius. The query I am using at the moment is:

SELECT  id, x, y
FROM    mytable t1
WHERE   (
        SELECT  COUNT(*)
        FROM    mytable t2
        WHERE   ABS(t1.x - t2.x) < 25
                AND ABS(t1.y - t2.y) < 25
        ) = 1


This is taking 15 minutes to run at times.

Is there a better way?

Of course using spatial abilities would be a better way, but it is possible to make do with plain SQL. This will also work in SQL Server 2005.

In most database engines, the spatial indexes are implemented as the R-Tree structures. SQL Server, however, uses another approach: surface tesselation.

Basically, it divides the surface into a finite number of tiles, each assigned with a unique number. The identifiers of tiles covered by the object are stored as keys of a plain B-Tree index.

When SQL Server‘s optimizer sees a geometrical predicate against an indexed column, it calculates the numbers of tiles that possibly can satisfy this predicate. Say, if the tiles are defined as squares with side 1, the predicate column.STDistance(@mypoint) < 2 can only be satisfied by the objects within 2 tiles away from @mypoint‘s tile. This gives a square of 25 tiles with @mypoint‘s tile in the center. The tile numbers can be found and searched for using the index. Exact filtering condition is then applied to each candidate value returned by the index.

Same solution can be used in our case even without the spatial functions. Comparing tile numbers is an equijoin and hash join method is eligible for this operation. We can even choose the tiling algorithm individually for each query, since we don’t have to store the tile identifiers in the table, and the hash table will be built dynamically anyway.

Let’s create a sample table and see how it works:

CREATE SCHEMA [20100226_circle]
CREATE TABLE t_circle (
        id INT NOT NULL PRIMARY KEY,
        x DECIMAL (15, 3) NOT NULL,
        y DECIMAL (15, 3) NOT NULL,
        )
GO
BEGIN TRANSACTION
SELECT  RAND(20100226)
DECLARE @cnt INT
SET @cnt = 1
WHILE @cnt <= 50000
BEGIN
        INSERT
        INTO    [20100226_circle].t_circle (id, x, y)
        VALUES  (
                @cnt,
                RAND() * 3200,
                RAND() * 3200
                )
        SET @cnt = @cnt + 1
END
COMMIT
GO

The table contains 50,000 points on random places within a square of 3,200 × 3,200 units.

We can optimize the original query a little by using NOT EXISTS instead of COUNT(*):

SELECT  *
FROM    [20100226_circle].t_circle co
WHERE   NOT EXISTS
        (
        SELECT  NULL
        FROM    [20100226_circle].t_circle ci
        WHERE   SQRT(POWER(co.x - ci.x, 2) + POWER(co.y - ci.y, 2)) < 25
                AND co.id <> ci.id
        )
ORDER BY
        id

View query results

, but this would still be quite slow.

The problem is that the nested loops go nowhere: they are still inside the plan, but return earlier. As a result, the query takes 5 minutes instead of 15, which is still too much.

To improve the query we need to make an efficient anti-join method to work, and the tesselation strategy is a way to go. Here’s what we need to do to implement this strategy:

  • Tesselate the surface and assign a unique number to each tile. Since we need to search for the records within 25 units, it will be a reasonable idea to divide the surface into a number of squares 25 × 25 units in size, numbered column-wise. To find out the number of rows and columns, we should just find the MIN and MAX x and y.

    This is the sample tesselation, assuming that MIN(x) and MIN(y) are 25 × 100 = 2500 units apart (and same with y).

  • Find the tile each point belongs to.

  • Build a recordset that would correspond each point to each of the tiles the neighbors can theoretically reside in. Since a circle with radius of 25 units theoretically may cover up to 9 adjacent tiles, each tile should be corresponded to each of the 9 tiles forming a 3 × 3 square with a unit’s tile in the center.

  • For each point, make sure that no other points exists within the neighboring tiles, additionally applying a fine-filtering condition.

    This may sound redundant, since the point-neighbor combination is unique, as well as point-tile, so only one of the 9 candidates will satisfy the join, even if the points reside in the adjacent tiles. But finding the exact distance requires a complex expression while comparing the tile numbers is an equality correlation and as such is eligible by an efficient anti-join method like HASH ANTI JOIN. The coarse filtering on tiles will sieve out most of the far neighbors so that the only adjacent neighbors will require special attention.

And here’s the query:

WITH    extremes AS
        (
        SELECT  *,
                maxx - minx AS width,
                maxy - miny AS height
        FROM    (
                SELECT  FLOOR(MIN(x) / 25) AS minx,
                        CEILING(MAX(x) / 25) AS maxx,
                        FLOOR(MIN(y) / 25) AS miny,
                        CEILING(MAX(y) / 25) AS maxy
                FROM    [20100226_circle].t_circle
                ) q
        ),
        tileset (dim) AS
        (
        SELECT  -1
        UNION ALL
        SELECT  0
        UNION ALL
        SELECT  1
        ),
        tiles AS
        (
        SELECT  id, x, y, minx, miny, width,
                (FLOOR(x / 25) - minx) * width + FLOOR(y / 25) - miny AS tile
        FROM    extremes
        CROSS JOIN
                [20100226_circle].t_circle
        ),
        neighbors AS
        (
        SELECT  ti.*,
                (FLOOR(ti.x / 25) - ti.minx + nx.dim) * ti.width +
                FLOOR(ti.y / 25) - ti.miny + ny.dim AS mtile
        FROM    tiles ti
        CROSS JOIN
                tileset nx
        CROSS JOIN
                tileset ny
        )
SELECT  *
FROM    tiles tn
WHERE   NOT EXISTS
        (
        SELECT  NULL
        FROM    neighbors n
        WHERE   n.mtile = tn.tile
                AND n.id <> tn.id
                AND SQRT(SQUARE(n.x - tn.x) + SQUARE(n.y - tn.y)) < 25
        )
ORDER BY
        id
id x y minx miny width tile
205 2247.896 3198.399 0 0 128 11519
2867 2159.626 1120.590 0 0 128 11052
13644 4.951 3165.734 0 0 128 126
15917 2747.826 3041.280 0 0 128 14073
25183 1858.866 326.416 0 0 128 9485
43211 1176.369 98.079 0 0 128 6019
6 rows fetched in 0.0017s (3.0781s) 
Table 't_circle'. Scan count 11, logical reads 3087, physical reads 0, read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.
Table 'Worktable'. Scan count 2, logical reads 207406, physical reads 0, read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.
Table 'Worktable'. Scan count 0, logical reads 0, physical reads 0, read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0. 

SQL Server Execution Times:
   CPU time = 4235 ms,  elapsed time = 3068 ms.

  |--Parallelism(Gather Streams, ORDER BY:([test].[20100226_circle].[t_circle].[id] ASC))
       |--Sort(ORDER BY:([test].[20100226_circle].[t_circle].[id] ASC))
            |--Compute Scalar(DEFINE:([Expr1007]=floor([Expr1003]/(25.)), [Expr1009]=floor([Expr1005]/(25.)), [Expr1011]=ceiling([Expr1004]/(25.))-floor([Expr1003]/(25.)), [Expr1016]=(([Expr1044]-floor([Expr1003]/(25.)))*(ceiling([Expr1004]/(25.))-floor([Expr1003]/(25.)))+[Expr1045])-floor([Expr1005]/(25.))))
                 |--Hash Match(Left Anti Semi Join, HASH:([Expr1055])=([Expr1054]), RESIDUAL:([Expr1054]=[Expr1055] AND [test].[20100226_circle].[t_circle].[id]<>[test].[20100226_circle].[t_circle].[id] AND sqrt(square(CONVERT_IMPLICIT(float(53),[test].[20100226_circle].[t_circle].[x]-[test].[20100226_circle].[t_circle].[x],0))+square(CONVERT_IMPLICIT(float(53),[test].[20100226_circle].[t_circle].[y]-[test].[20100226_circle].[t_circle].[y],0)))<(2.500000000000000e+001)))
                      |--Bitmap(HASH:([Expr1055]), DEFINE:([Bitmap1056]))
                      |    |--Parallelism(Repartition Streams, Hash Partitioning, PARTITION COLUMNS:([Expr1055]))
                      |         |--Compute Scalar(DEFINE:([Expr1055]=(([Expr1044]-floor([Expr1003]/(25.)))*(ceiling([Expr1004]/(25.))-floor([Expr1003]/(25.)))+[Expr1045])-floor([Expr1005]/(25.))))
                      |              |--Nested Loops(Inner Join)
                      |                   |--Parallelism(Distribute Streams, Broadcast Partitioning)
                      |                   |    |--Stream Aggregate(DEFINE:([Expr1003]=MIN([partialagg1048]), [Expr1004]=MAX([partialagg1049]), [Expr1005]=MIN([partialagg1050])))
                      |                   |         |--Parallelism(Gather Streams)
                      |                   |              |--Stream Aggregate(DEFINE:([partialagg1048]=MIN([test].[20100226_circle].[t_circle].[x]), [partialagg1049]=MAX([test].[20100226_circle].[t_circle].[x]), [partialagg1050]=MIN([test].[20100226_circle].[t_circle].[y])))
                      |                   |                   |--Clustered Index Scan(OBJECT:([test].[20100226_circle].[t_circle].[PK__t_circle__62065FF3]))
                      |                   |--Compute Scalar(DEFINE:([Expr1044]=floor([test].[20100226_circle].[t_circle].[x]/(25.)), [Expr1045]=floor([test].[20100226_circle].[t_circle].[y]/(25.))))
                      |                        |--Clustered Index Scan(OBJECT:([test].[20100226_circle].[t_circle].[PK__t_circle__62065FF3]))
                      |--Parallelism(Repartition Streams, Hash Partitioning, PARTITION COLUMNS:([Expr1054]), WHERE:(PROBE([Bitmap1056])=TRUE))
                           |--Compute Scalar(DEFINE:([Expr1054]=(((([Expr1046]-floor([Expr1020]/(25.)))+CONVERT_IMPLICIT(decimal(10,0),[Union1037],0))*(ceiling([Expr1021]/(25.))-floor([Expr1020]/(25.)))+[Expr1047])-floor([Expr1022]/(25.)))+CONVERT_IMPLICIT(decimal(10,0),[Union1041],0)))
                                |--Nested Loops(Inner Join)
                                     |--Nested Loops(Inner Join)
                                     |    |--Parallelism(Distribute Streams, RoundRobin Partitioning)
                                     |    |    |--Nested Loops(Inner Join)
                                     |    |         |--Stream Aggregate(DEFINE:([Expr1020]=MIN([partialagg1051]), [Expr1021]=MAX([partialagg1052]), [Expr1022]=MIN([partialagg1053])))
                                     |    |         |    |--Parallelism(Gather Streams)
                                     |    |         |         |--Stream Aggregate(DEFINE:([partialagg1051]=MIN([test].[20100226_circle].[t_circle].[x]), [partialagg1052]=MAX([test].[20100226_circle].[t_circle].[x]), [partialagg1053]=MIN([test].[20100226_circle].[t_circle].[y])))
                                     |    |         |              |--Clustered Index Scan(OBJECT:([test].[20100226_circle].[t_circle].[PK__t_circle__62065FF3]))
                                     |    |         |--Constant Scan(VALUES:(((-1)),((0)),((1))))
                                     |    |--Constant Scan(VALUES:(((-1)),((0)),((1))))
                                     |--Table Spool
                                          |--Compute Scalar(DEFINE:([Expr1046]=floor([test].[20100226_circle].[t_circle].[x]/(25.)), [Expr1047]=floor([test].[20100226_circle].[t_circle].[y]/(25.))))
                                               |--Clustered Index Scan(OBJECT:([test].[20100226_circle].[t_circle].[PK__t_circle__62065FF3]))

This query only takes 3 seconds.

Hope that helps.

I’m always glad to answer the questions regarding database queries.

Ask me a question

]]> http://explainextended.com/2010/02/26/efficient-circle-distance-testing/feed/ 0 SQL Server: EXCEPT ALL http://explainextended.com/2010/02/10/sql-server-except-all/ http://explainextended.com/2010/02/10/sql-server-except-all/#comments Wed, 10 Feb 2010 20:00:52 +0000 Quassnoi http://explainextended.com/?p=4270 Answering questions asked on the site.

myst asks:

I have two really large tables with lots of columns, many of them are nullable

I need to remove all rows from table1 which are not present in table2, but there can be duplicates in both tables and writing more than 70 IS NULL conditions would be a pain, and I want to make sure there’s nothing I’m missing.

Is there a more simple way?

SQL Server supports EXCEPT clause which returns all records present in the first table and absent in the second one. But this clause eliminates duplicates and cannot be used as a subject to a DML operation.

ANSI SQL standard describes EXCEPT ALL which returns all records from the first table which are not present in the second table, leaving the duplicates as is. Unfortunately, SQL Server does not support this operator.

Similar behavior can be achieved using NOT IN or NOT EXISTS constructs. But in SQL Server, IN predicate does not accept more than one field. NOT EXISTS accepts any number of correlated columns, but it requires extra checks in the WHERE clause, since equality operator does not treat two NULL values as equal. Each pair or nullable columns should be additionally checked for a NULL in both fields. This can only be done using OR predicates or COALESCE, neither of which adds to performance.

But there is a way to emulate EXCEPT ALL in SQL Server quite elegantly and efficiently.

Let’s create two sample tables:

Table creation details

The second table is 20 records short of being the full copy of the first table. There are NULL values and duplicates in both tables.

The EXCEPT query returns this:

SELECT  *
FROM    [20100211_except].t1 AS t1
EXCEPT
SELECT  *
FROM    [20100211_except].t2 AS t2
val1 val2 val3 val4 val5
9999 9999 9999 9999
9998 9998 9998 9998 9998
9998 9998 9998 9998
9999 9999 9999 9999 9999
4 rows fetched in 0.0011s (0.1933s)

View query details

There are only 4 records in the resultset, since the duplicates are eliminated. And you cannot delete from this resultset.

Here’s the EXISTS workaround:

SELECT  *
FROM    [20100211_except].t1 AS t1
WHERE   NOT EXISTS
        (
        SELECT  NULL
        FROM    [20100211_except].t2 AS t2
        WHERE   1 = 1
                AND (t1.val1 = t2.val1 OR (t1.val1 IS NULL AND t2.val1 IS NULL))
                AND (t1.val2 = t2.val2 OR (t1.val2 IS NULL AND t2.val2 IS NULL))
                AND (t1.val3 = t2.val3 OR (t1.val3 IS NULL AND t2.val3 IS NULL))
                AND (t1.val4 = t2.val4 OR (t1.val4 IS NULL AND t2.val4 IS NULL))
                AND (t1.val5 = t2.val5 OR (t1.val5 IS NULL AND t2.val5 IS NULL))
        )
val1 val2 val3 val4 val5
9999 9999 9999 9999 9999
9999 9999 9999 9999 9999
9999 9999 9999 9999 9999
9999 9999 9999 9999 9999
9999 9999 9999 9999 9999
9998 9998 9998 9998 9998
9998 9998 9998 9998
9998 9998 9998 9998 9998
9998 9998 9998 9998
9998 9998 9998 9998 9998
9998 9998 9998 9998
9998 9998 9998 9998 9998
9998 9998 9998 9998
9998 9998 9998 9998 9998
9998 9998 9998 9998
9999 9999 9999 9999
9999 9999 9999 9999
9999 9999 9999 9999
9999 9999 9999 9999
9999 9999 9999 9999
20 rows fetched in 0.0029s (0.1953s)

View query details

This returns all records, but you just look at that query. And it’s only 5 columns, not 70.

To emulate EXCEPT ALL, we can use a little trick.

Unlike PostgreSQL, SQL Server does not support record types. You cannot return a whole record in one field. But it is possible to return a whole record in a tableless SELECT in the correlated subquery. And plain EXCEPT operator can be applied to this SELECT.

Using this trick, EXCEPT may be applied to each individual record from the first table, not to the table as a whole. If this record is presented among the other table’s records, nothing will be returned by the operator; otherwise a single row which was initially supplied will be returned. This can be filtered using EXISTS clause.

Here’s how it looks:

SELECT  *
FROM    [20100211_except].t1 AS t1
WHERE   EXISTS
        (
        SELECT  t1.*
        EXCEPT
        SELECT  *
        FROM    [20100211_except].t2 AS t2
        )
val1 val2 val3 val4 val5
9998 9998 9998 9998 9998
9998 9998 9998 9998
9998 9998 9998 9998 9998
9998 9998 9998 9998
9998 9998 9998 9998 9998
9998 9998 9998 9998
9998 9998 9998 9998 9998
9998 9998 9998 9998
9998 9998 9998 9998 9998
9998 9998 9998 9998
9999 9999 9999 9999 9999
9999 9999 9999 9999
9999 9999 9999 9999 9999
9999 9999 9999 9999
9999 9999 9999 9999 9999
9999 9999 9999 9999
9999 9999 9999 9999 9999
9999 9999 9999 9999
9999 9999 9999 9999 9999
9999 9999 9999 9999
20 rows fetched in 0.0033s (0.7656s)

View query details

All 20 records are on their places.

Note that this query didn’t use any actual field names at all: only pure set-based operators.

This is a little bit less efficient than a normal field-based EXISTS, because this query is forced to use Nested Loops. However, with an index that is built in-place by the Eager Spool operation, this is still quite fast.

This query behaves just like EXCEPT ALL but formally, it’s a single-table SELECT query without joins or aggregations. And as such, it can be used in the DML.

Use this query to delete all records absent in the second table:

DELETE  t1
FROM    [20100211_except].t1 AS t1
WHERE   EXISTS
        (
        SELECT  t1.*
        EXCEPT
        SELECT  *
        FROM    [20100211_except].t2
        )

, and this one to delete records present in the second table:

DELETE  t1
FROM    [20100211_except].t1 AS t1
WHERE   NOT EXISTS
        (
        SELECT  t1.*
        EXCEPT
        SELECT  *
        FROM    [20100211_except].t2
        )

Beware: EXISTS predicate returns absent records, and NOT EXISTS returns present records.

This is not very intuitive, but may become more clear if you realize that the subquery effectively checks for absence of the current row, so the EXISTS predicate in fact says where exists the absence of the current row.

Hope that helps.

Update of Feb 12th, 2009

In a feedback, Gonçalo Ferreira suggested using INTERSECT instead of EXCEPT in the inner subquery:

Wouldn’t using INTERSECT in the sub-query avoid the EXISTS – absent / NOT EXISTS – present confusion?

Let’s try it:

SELECT  *
FROM    [20100211_except].t1 AS t1
WHERE   NOT EXISTS
        (
        SELECT  t1.*
        INTERSECT
        SELECT  *
        FROM    [20100211_except].t2 AS t2
        )
val1 val2 val3 val4 val5
9999 9999 9999 9999 9999
9999 9999 9999 9999 9999
9999 9999 9999 9999 9999
9999 9999 9999 9999 9999
9999 9999 9999 9999 9999
9998 9998 9998 9998 9998
9998 9998 9998 9998
9998 9998 9998 9998 9998
9998 9998 9998 9998
9998 9998 9998 9998 9998
9998 9998 9998 9998
9998 9998 9998 9998 9998
9998 9998 9998 9998
9998 9998 9998 9998 9998
9998 9998 9998 9998
9999 9999 9999 9999
9999 9999 9999 9999
9999 9999 9999 9999
9999 9999 9999 9999
9999 9999 9999 9999
20 rows fetched in 0.0027s (0.1754s)

View query details

Not only this is more readable, but this is also much faster, since it uses a single Hash Match (Right Anti Semi Join) (which is the same thing as the equality EXISTS clause used).

Nice point, Gonçalo!

I’m always glad to answer the questions regarding database queries.

Ask me a question

]]> http://explainextended.com/2010/02/10/sql-server-except-all/feed/ 0 SQL Server: running totals http://explainextended.com/2010/01/22/sql-server-running-totals/ http://explainextended.com/2010/01/22/sql-server-running-totals/#comments Fri, 22 Jan 2010 20:00:18 +0000 Quassnoi http://explainextended.com/?p=4009 From Stack Overflow:

We have a table of transactions which is structured like the following :

transactions
TranxID ItemID TranxDate TranxAmt

TranxAmt can be positive or negative, so the running total of this field (for any ItemID) will go up and down as time goes by.

Getting the current total is obviously simple, but what I’m after is a performant way of getting the highest value of the running total and the TranxDate when this occurred.

Note that TranxDate is not unique.

SQL Server is a very nice system, but using it for calculating running totals is a pain.

Oracle supports additional clauses for analytic functions, RANGE and ROWS, which define the boundaries of the function’s windows and hence can be used to implement running totals. By default, it is just enough to omit the RANGE clause to make the analytic function apply to the window of the records selected so far, thus transforming it to a running total.

SQL Server‘s support for window functions only extends aggregate capabilities a little so that the aggregate can be returned along with each record that constitutes the group. For functions like SUM and COUNT it is impossible to control the window boundaries and the records order. Such analytic functions can not be used to calculate running totals.

The common way to write such a running total query is using a subquery or a self join which would count the SUM of all previous records. However, the complexity of this query is O(n^2) and it’s not usable for any real volumes of data.

This is one of the few cases when the cursors are faster than a set-based solution described above. But we all are aware of the drawbacks of cursors and better search for something else.

This task, fortunately, is a little more simple than it may seem, because it deals with dates. The number of all possible dates is usually limited and a recursive query can deal with this task quite efficiently.

Let’s create a sample table:

Table creation details

There are 100,000 records with 5 ItemID‘s (20,000 records per ItemID). The dates and the values are random, with 5,000 possible dates.

Now, the idea is to develop a recursive query that would iterate though dates and on each date, add the sum of that date’s transactions.

Here’s how we do it (in steps):

Grouping the records

The first thing we need to go is group the records by ItemID and TransDate:

WITH    q AS
        (
        SELECT  ItemID, TranxDate, SUM(TranxAmt) AS TranxSum
        FROM    [20100122_running].t_transaction
        GROUP BY
                ItemID, TranxDate
        )

This kills two birds: first, this is a part of a query we will need to obtain the seed values to begin recursion with, second, we get a set of records that are unique on (ItemID, TranxDate) (this will be required a little bit later)

Retrieving the seed values

A recursive CTE should have an anchor part which is a base for recursion. To get the initial values of dates and sums we need to select the records holding the minimal values of TranxDate for each ItemID. The link above describes two approaches to that in more detail, and here I’ll just proved the query:

WITH    q AS
        (
        SELECT  ItemID, TranxDate, SUM(TranxAmt) AS TranxSum
        FROM    [20100122_running].t_transaction
        GROUP BY
                ItemID, TranxDate
        ),
        SELECT  qo.ItemID, LastDate, TranxDate, TranxSum
        FROM    (
                SELECT  ItemID, MAX(TranxDate) AS LastDate
                FROM    q
                GROUP BY
                        ItemID
                ) qo
        CROSS APPLY
                (
                SELECT  TOP 1 TranxDate, TranxSum
                FROM    q qi
                WHERE   qi.ItemID = qo.ItemId
                ORDER BY
                        TranxDate
                ) qn
        )

Note that I use GROUP BY instead of DISTINCT in the subquery that selects the ItemID‘s. This allows to select the value of MAX(TranxDate) along with each ItemID which could be used as a stop condition in the recursive CTE.

Calculating the running SUM

To calculate the running SUM we need to run the query recursively, adding one day to TranxDate until it reaches LastDate (which we selected with the previous query).

The value of the running SUM will be that of the previous day plus the total amount of transactions for the current day. Unfortunately, due to some limitations of SQL Server‘s implementation of recursive CTE‘s, we can use neither LEFT JOIN nor aggregate functions (like SUM) in a recursive part of the CTE.

That’s why we should use precalculated SUM‘s for the current (ItemID, TranxDate) (which is one of the reasons for pregrouping the records that I mentioned earlier in the article), and retrieve its value using a scalar subquery instead of a LEFT JOIN:

WITH    q AS
        (
        SELECT  ItemID, TranxDate, SUM(TranxAmt) AS TranxSum
        FROM    [20100122_running].t_transaction
        GROUP BY
                ItemID, TranxDate
        ),
        m AS
        (
        SELECT  qo.ItemID, LastDate, TranxDate, TranxSum
        FROM    (
                SELECT  ItemID, MAX(TranxDate) AS LastDate
                FROM    q
                GROUP BY
                        ItemID
                ) qo
        CROSS APPLY
                (
                SELECT  TOP 1 TranxDate, TranxSum
                FROM    q qi
                WHERE   qi.ItemID = qo.ItemId
                ORDER BY
                        TranxDate
                ) qn
        UNION ALL
        SELECT  ItemId, LastDate, DATEADD(day, 1, m.TranxDate),
                m.TranxSum +
                COALESCE(
                (
                SELECT  TranxSum
                FROM    q
                WHERE   q.ItemID = m.ItemId
                        AND q.TranxDate = DATEADD(day, 1, m.TranxDate)
                ), 0)
        FROM    m
        WHERE   m.TranxDate < m.LastDate
        )

Since there are neither a LEFT JOIN nor a SUM in the recursive part of the query, SQL Server allows this construct.

If we were to SELECT from this CTE we would get a list of running sums for each ItemID.

Selecting greatest running sums

All we have to do now is to select records holding a greatest running sum per ItemID.

This article (which I already mentioned above) describes two methods for doing this in SQL Server.

We already used one of these methods, the one which uses DISTINCT / CROSS APPLY. For that task it was better since we needed a GROUP BY anyway to get the LastDate.

However, for getting the records holding greatest running sums, we will use another method, which just filters on partitioned ROW_NUMBER. This method is better, since the query in question is quite a complex CTE. SQL Server does not usually materialize the CTE‘s and reevaluating it each time in a CROSS APPLY would be quite expensive.

The ROW_NUMBER, on the contrary, can be retrieved and filtered on with a single pass over the CTE‘s resultset.

Here’s how we do it:

WITH    q AS
        (
        SELECT  ItemID, TranxDate, SUM(TranxAmt) AS TranxSum
        FROM    [20100122_running].t_transaction
        GROUP BY
                ItemID, TranxDate
        ),
        m AS
        (
        SELECT  qo.ItemID, LastDate, TranxDate, TranxSum
        FROM    (
                SELECT  ItemID, MAX(TranxDate) AS LastDate
                FROM    q
                GROUP BY
                        ItemID
                ) qo
        CROSS APPLY
                (
                SELECT  TOP 1 TranxDate, TranxSum
                FROM    q qi
                WHERE   qi.ItemID = qo.ItemId
                ORDER BY
                        TranxDate
                ) qn
        UNION ALL
        SELECT  ItemId, LastDate, DATEADD(day, 1, m.TranxDate),
                m.TranxSum +
                COALESCE(
                (
                SELECT  TranxSum
                FROM    q
                WHERE   q.ItemID = m.ItemId
                        AND q.TranxDate = DATEADD(day, 1, m.TranxDate)
                ), 0)
        FROM    m
        WHERE   m.TranxDate < m.LastDate
        )
SELECT  ItemID, TranxDate, TranxSum
FROM    (
        SELECT  m.*, ROW_NUMBER() OVER (PARTITION BY ItemId ORDER BY TranxSum DESC, TranxDate) AS rn
        FROM    m
        ) qp
WHERE   rn = 1
OPTION (MAXRECURSION 0)
ItemID TranxDate TranxSum
1 1999-10-26 00:00:00.000 406.0530
2 2007-07-08 00:00:00.000 5594.7071
3 2009-06-04 00:00:00.000 4903.4597
4 1996-10-14 00:00:00.000 22.3472
5 2008-06-18 00:00:00.000 247.2859
5 rows fetched in 0.0004s (1.0937s) 
Table 'Worktable'. Scan count 2, logical reads 149994, physical reads 0, read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.
Table 't_transaction'. Scan count 49534, logical reads 151549, physical reads 0, read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0. 

SQL Server Execution Times:
   CPU time = 1016 ms,  elapsed time = 1101 ms.

  |--Filter(WHERE:([Expr1029]=(1)))
       |--Sequence Project(DEFINE:([Expr1029]=row_number))
            |--Compute Scalar(DEFINE:([Expr1041]=(1)))
                 |--Segment
                      |--Sort(ORDER BY:([Recr1025] ASC, [Recr1028] DESC, [Recr1027] ASC))
                           |--Index Spool(WITH STACK)
                                |--Concatenation
                                     |--Compute Scalar(DEFINE:([Expr1036]=(0)))
                                     |    |--Nested Loops(Inner Join, OUTER REFERENCES:([test].[20100122_running].[t_transaction].[ItemID]))
                                     |         |--Stream Aggregate(GROUP BY:([test].[20100122_running].[t_transaction].[ItemID]) DEFINE:([Expr1004]=MAX([test].[20100122_running].[t_transaction].[TranxDate])))
                                     |         |    |--Stream Aggregate(GROUP BY:([test].[20100122_running].[t_transaction].[ItemID], [test].[20100122_running].[t_transaction].[TranxDate]))
                                     |         |         |--Index Scan(OBJECT:([test].[20100122_running].[t_transaction].[IX_transaction_date__amt]), ORDERED FORWARD)
                                     |         |--Top(TOP EXPRESSION:((1)))
                                     |              |--Stream Aggregate(GROUP BY:([test].[20100122_running].[t_transaction].[TranxDate]) DEFINE:([Expr1008]=SUM([test].[20100122_running].[t_transaction].[TranxAmt])))
                                     |                   |--Index Seek(OBJECT:([test].[20100122_running].[t_transaction].[IX_transaction_date__amt]), SEEK:([test].[20100122_running].[t_transaction].[ItemID]=[test].[20100122_running].[t_transaction].[ItemID]) ORDERED FORWARD)
                                     |--Nested Loops(Inner Join, OUTER REFERENCES:([Expr1038], [Recr1009], [Recr1010], [Recr1011], [Recr1012]))
                                          |--Compute Scalar(DEFINE:([Expr1038]=[Expr1037]+(1)))
                                          |    |--Table Spool(WITH STACK)
                                          |--Compute Scalar(DEFINE:([Expr1013]=dateadd(day,(1),[Recr1011]), [Expr1024]=[Recr1012]+CASE WHEN [Expr1017] IS NOT NULL THEN [Expr1022] ELSE ($0.0000) END))
                                               |--Nested Loops(Left Outer Join, PASSTHRU:(IsFalseOrNull [Expr1017] IS NOT NULL))
                                                    |--Nested Loops(Left Outer Join)
                                                    |    |--Filter(WHERE:(STARTUP EXPR([Recr1011]<[Recr1010])))
                                                    |    |    |--Constant Scan
                                                    |    |--Stream Aggregate(DEFINE:([Expr1017]=SUM([test].[20100122_running].[t_transaction].[TranxAmt])))
                                                    |         |--Index Seek(OBJECT:([test].[20100122_running].[t_transaction].[IX_transaction_date__amt]), SEEK:([test].[20100122_running].[t_transaction].[ItemID]=[Recr1009] AND [test].[20100122_running].[t_transaction].[TranxDate]=dateadd(day,(1),[Recr1011])) ORDERED FORWARD)
                                                    |--Stream Aggregate(DEFINE:([Expr1022]=SUM([test].[20100122_running].[t_transaction].[TranxAmt])))
                                                         |--Index Seek(OBJECT:([test].[20100122_running].[t_transaction].[IX_transaction_date__amt]), SEEK:([test].[20100122_running].[t_transaction].[ItemID]=[Recr1009] AND [test].[20100122_running].[t_transaction].[TranxDate]=dateadd(day,(1),[Recr1011])) ORDERED FORWARD)

The query completes in 1 second which is very fast.

It performs much better than a cursor-based solution and of course very much better than SELECT-level subqueries or self-joins that calculate the same sums of the same rows over and over and over again.

]]> http://explainextended.com/2010/01/22/sql-server-running-totals/feed/ 1 Calculating mode http://explainextended.com/2010/01/18/calculating-mode/ http://explainextended.com/2010/01/18/calculating-mode/#comments Mon, 18 Jan 2010 20:00:32 +0000 Quassnoi http://explainextended.com/?p=3987 From Stack Overflow:

I have this query:

WITH    CTE AS (
        SELECT  e_id,
                scale,
                ROW_NUMBER() OVER(PARTITION BY e_id ORDER BY scale ASC) AS rn,
                COUNT(scale) OVER(PARTITION BY e_id) AS cn
        FROM    ScoreMaster
        WHERE   scale IS NOT NULL
        )
SELECT  e_id,
        AVG(scale) AS [AVG],
        STDEV(scale) AS [StdDev],
        AVG(CASE WHEN 2 * rn - cn BETWEEN 0 AND 2 THEN scale END) AS [FinancialMedian]
        MAX(CASE WHEN 2 * rn - cn BETWEEN 0 AND 2 THEN scale END) AS [StatisticalMedian]
FROM    CTE
GROUP BY
        e_id

How do I add Mode to this query?

A quick reminder: in statistics, mode is the value that occurs most frequently in a data set.

In other words, for each e_id, mode is the (exact) value of scale shared by most records with this e_id.

Unlike other statistical parameters used in this query, mode is not guaranteed to have a single value. If, say, 10 records have scale = 1 and 10 other records have scale = 2 (and all other values of scale are shared by less than 10 records), then there are two modes in this set (and the set, hence, is called bimodal). Likewise, there can be trimodal, quadrimodal or, generally speaking, multimodal sets.

This means that we should define a way on how to choose this mode.

There can be three approaches to this:

  1. Return every modal value
  2. Return a single modal value
  3. Return an aggregate of all modal values

To check all queries, we will generate a simple trimodal dataset:

Table creation details

The basic idea is simple: we should find a value that is held by the maximum number or records. To do this, we need to calculate the number of records sharing a given value of scale. We do this by adding a COUNT as an analytical function into the CTE:

WITH    cte AS
        (
        SELECT  e_id,
                scale,
                ROW_NUMBER() OVER (PARTITION BY e_id ORDER BY scale ASC) AS rn,
                COUNT(scale) OVER (PARTITION BY e_id) AS cn,
                COUNT(*) OVER (PARTITION BY e_id, scale) AS sn
        FROM    [20100118_mode].ScoreMaster
        WHERE   scale IS NOT NULL
        )

Single value

In this case, we just pick a minimal modal value.

To do this, we group the results returned by e_id (just like the original query does) and select the TOP 1 scale ordered by sn DESC, scale in a subquery:

WITH    cte AS
        (
        SELECT  e_id,
                scale,
                ROW_NUMBER() OVER (PARTITION BY e_id ORDER BY scale ASC) AS rn,
                COUNT(scale) OVER (PARTITION BY e_id) AS cn,
                COUNT(*) OVER (PARTITION BY e_id, scale) AS sn
        FROM    [20100118_mode].ScoreMaster
        WHERE   scale IS NOT NULL
        )
SELECT  e_id,
        (
        SELECT  TOP 1 scale
        FROM    cte ci
        WHERE   ci.e_id = co.e_id
        ORDER BY
                sn DESC, scale
        )
FROM    cte co
GROUP BY
        e_id
e_id
1 .2000

To select the maximal modal value instead of the minimal one, we would need to add a DESC to the score in the ORDER BY clause.

By picking a right final part of the ORDER BY clause, any other value can be returned.

However, the first part of the ORDER BY (sn DESC) should always remain the same, since it’s what makes a modal value to be selected first.

Average of all modal values

To select the average of all modal values, we need to calculate it in the subquery as well.

We can use a very interesting predicate here:

WITH    cte AS
        (
        SELECT  e_id,
                scale,
                ROW_NUMBER() OVER (PARTITION BY e_id ORDER BY scale ASC) AS rn,
                COUNT(scale) OVER (PARTITION BY e_id) AS cn,
                COUNT(*) OVER (PARTITION BY e_id, scale) AS sn
        FROM    [20100118_mode].ScoreMaster
        WHERE   scale IS NOT NULL
        )
SELECT  e_id,
        (
        SELECT  AVG(scale)
        FROM    cte ci
        WHERE   ci.e_id = co.e_id
                AND ci.scale = MAX(co.scale)
        )
FROM    cte co
GROUP BY
        e_id
e_id
1 .500000

Note this line in the WHERE clause:

                AND ci.scale = MAX(co.scale)

This is a naive approach most beginner developers try to use to select records holding a maximal value of the column.

This of course never works for this purpose (because WHERE clause is evaluated before the aggregation). In our case, though, we use MAX as a reference value in the SELECT-level correlated subquery, with the aggregation already performed. So the value of the MAX (taken from results of the other query) can be used in the WHERE clause alright.

AVG in the subquery can be replaced by any other aggregate.

If replaced by MIN or MAX, this aggregate variant becomes a synonym for the first query (which selects a single value). However, the first approach is more efficient for that.

Returning all modes

To return all modal values, we should use CROSS APPLY clause, and return all DISTINCT scales held by the maximal number of records.

Since it’s performed before GROUP BY, we will need to wrap the original query into the subquery (so that GROUP BY is performed first). This way, we can also calculate the MAX(sn) to use it in the CROSS APPLY later:

WITH    cte AS
        (
        SELECT  e_id,
                scale,
                ROW_NUMBER() OVER (PARTITION BY e_id ORDER BY scale ASC) AS rn,
                COUNT(scale) OVER (PARTITION BY e_id) AS cn,
                COUNT(*) OVER (PARTITION BY e_id, scale) AS sn
        FROM    [20100118_mode].ScoreMaster
        WHERE   scale IS NOT NULL
        )
SELECT  e_id, scale
FROM    (
        SELECT  e_id, MAX(sn) AS msn
        FROM    cte
        GROUP BY
                e_id
        ) co
CROSS APPLY
        (
        SELECT  DISTINCT scale
        FROM    cte ci
        WHERE   ci.e_id = co.e_id
                AND ci.sn = co.msn
        ) ci
e_id scale
1 .2000
1 .3000
1 .4000

This query, unlike previous two, returns several records per e_id, each holding a modal value of scale.

]]> http://explainextended.com/2010/01/18/calculating-mode/feed/ 0 Grouping continuous ranges http://explainextended.com/2009/12/30/grouping-continuous-ranges/ http://explainextended.com/2009/12/30/grouping-continuous-ranges/#comments Wed, 30 Dec 2009 20:00:32 +0000 Quassnoi http://explainextended.com/?p=3904 From Stack Overflow:

I have a ticketing system.

Now I have to select adjacent places when the user asks for 2 or 3 tickets.

Every ticket has a line and column number. The concept of adjacent places is places in the same line with adjacent columns numbers.

These tickets are in an SQL Server database.

Any ideas about this algorithm to search for available adjacent seats?

This is a problem known as grouping continuous ranges: finding the continuous ranges of the records (in a certain order) having the same value of the grouping column.

For the table in question, the groups will look like this:

Seats
Row Column Occupied Group
1 1 1 -
1 2 0 1
1 3 0 1
1 4 0 1
1 5 1 -
1 6 1 -
1 7 0 2
1 8 0 2
1 9 1 -
1 10 0 3

To find the spans of free seats with the required length, we need to group the continuous ranges of rows with occupied = 0 in column order (within the rows), calculate the counts of records in these groups and return the groups having sufficient value of COUNT(*).

To group something in SQL, we need to have an expression that would have the same value for all records belonging to the group.

How do we build such an expression for the continuous ranges?

In a movie theater, seats are numbered sequentially, so we can rely on the values of the column being continuous. Now, let’s calculate only the records that describe the free seats, and build their row numbers in the column order:

Seats
Row Column Occupied ROW_NUMBER
1 2 0 1
1 3 0 2
1 4 0 3
1 7 0 4
1 8 0 5
1 10 0 6

We see that filtering only free seats broke the column numbering order (the columns numbers are not continuous anymore), but the ROW_NUMBERs are continuous (by definition). Each occupied seat breaks the continuity of the columns, and the spans of the free seats correspond to the ranges of columns with unbroken continuity.

If the continuity is not broken, column and ROW_NUMBER both increase by 1 with each record. This means that their difference will be constant:

Seats
Row Column Occupied ROW_NUMBER column – ROW_NUMBER
1 2 0 1 1
1 3 0 2 1
1 4 0 3 1
1 7 0 4 3
1 8 0 5 3
1 10 0 6 4

The value of difference between column and ROW_NUMBER, therefore, uniquely defines the continuous group the records belong to.

This means we can group on it and do other things (the rest being pure technical).

Let’s create a sample table:

Table creation details

and select the chart of the places:

SELECT  row, [1], [2], [3], [4], [5], [6], [7], [8], [9], [10]
FROM    (
        SELECT  row, col,
                CASE occupied WHEN 0 THEN ' ' ELSE 'X' END AS occupied
        FROM    [20091230_seats].t_seat
        ) q
PIVOT   (
        MAX(occupied)
        FOR col IN
        ([1], [2], [3], [4], [5], [6], [7], [8], [9], [10])
        ) p
row 1 2 3 4 5 6 7 8 9 10
1 X X X
2 X X X X
3 X X X X X X
4 X X X X
5 X X X
6 X X X X X X
7 X X
8 X X X X
9 X X X X X
10 X X X X X
10 rows fetched in 0.0011s (0.0012s)

Now, let’s select the seat spans having length 5 or more:

WITH    free AS
        (
        SELECT  row, col, ROW_NUMBER() OVER (PARTITION BY row ORDER BY col) AS rn
        FROM    [20091230_seats].t_seat
        WHERE   occupied = 0
        ),
        spans AS
        (
        SELECT  row, col, COUNT(*) OVER (PARTITION BY row, col - rn) AS rl
        FROM    free
        )
SELECT  *
FROM    spans
WHERE   rl >= 5
row col rl
5 2 6
5 3 6
5 4 6
5 5 6
5 6 6
5 7 6
7 4 7
7 5 7
7 6 7
7 7 7
7 8 7
7 9 7
7 10 7
13 rows fetched in 0.0005s (0.0016s)

and put them on the chart:

WITH    free AS
        (
        SELECT  row, col, ROW_NUMBER() OVER (PARTITION BY row ORDER BY col) AS rn
        FROM    [20091230_seats].t_seat
        WHERE   occupied = 0
        ),
        spans AS
        (
        SELECT  row, col, COUNT(*) OVER (PARTITION BY row, col - rn) AS rl
        FROM    free
        )
SELECT  row, [1], [2], [3], [4], [5], [6], [7], [8], [9], [10]
FROM    (
        SELECT  se.row, se.col,
                CASE
                WHEN rl IS NULL THEN
                        'X'
                WHEN rl < 5 THEN
                        ' '
                ELSE
                        CAST(rl AS VARCHAR)
                END AS occupied
        FROM    [20091230_seats].t_seat se
        LEFT JOIN
                spans sp
        ON      sp.row = se.row
                AND sp.col = se.col
        ) q
PIVOT   (
        MAX(occupied)
        FOR col IN
        ([1], [2], [3], [4], [5], [6], [7], [8], [9], [10])
        ) p
row 1 2 3 4 5 6 7 8 9 10
1 X X X
2 X X X X
3 X X X X X X
4 X X X X
5 X 6 6 6 6 6 6 X X
6 X X X X X X
7 X X 7 7 7 7 7 7 7
8 X X X X
9 X X X X X
10 X X X X X
10 rows fetched in 0.0012s (0.0098s)

Free spans of 5 seats or more (in rows 5 and 7) are marked on the chart with the numbers corresponding to the number of free seats in the spans.

]]> http://explainextended.com/2009/12/30/grouping-continuous-ranges/feed/ 0 SQL Server: Selecting records holding group-wise maximum (with ties) http://explainextended.com/2009/12/01/sql-server-selecting-records-holding-group-wise-maximum-with-ties/ http://explainextended.com/2009/12/01/sql-server-selecting-records-holding-group-wise-maximum-with-ties/#comments Tue, 01 Dec 2009 20:00:58 +0000 Quassnoi http://explainextended.com/?p=3822 A feedback on the yesterday’s article on selecting records holding group-wise maximums in SQL Server.

Gonçalo asks:

Regarding the post I mention on the subject, wouldn’t it be easier to obtain the result you’re after using the SQL Server specific SELECT TOP x WITH TIES?

This clause (which is indeed specific to SQL Server) returns ties. TOP x WITH TIES means return TOP x records, plus all records holding the same values of the ORDER BY expressions as the last (x‘th) record returned by the TOP.

With a little effort this can be used for one of the queries I wrote about yesterday, namely the one that returns all records holding MIN(orderer) within each group.

However, the efficiency of this solution varies greatly depending on the cardinality of both ordering and grouping columns.

Let’s create a sample table:

Table creation details

Just like in the articles before that this table contains 1,000,000 records with the following fields:

  • id is the PRIMARY KEY
  • orderer is filled with random values from 1 to 10
  • glow is a low cardinality grouping field (10 distinct values)
  • ghigh is a high cardinality grouping field (10,000 distinct values)
  • stuffing is an asterisk-filled VARCHAR(200) column added to emulate payload of the actual tables

Additionally, there is one more field: lorderer. Just like orderer, it is filled with random values, but has much greater cardinality: the values are 1 to 10,000.

Now, let’s try to select the records holding MIN(orderer) using the two approaches.

Low cardinality orderers

Using DENSE_RANK

SELECT  COUNT(*) AS cnt, SUM(LEN(stuffing)) AS psum
FROM    (
        SELECT  d.*, DENSE_RANK() OVER (PARTITION BY glow ORDER BY orderer) AS dr
        FROM    [20091201_ties].t_distinct d
        ) dd
WHERE   dr = 1

View query details

As in the yesterday’s table, this query runs for 41 second. The plan used by the engine involves partitioning, sorting and gathering the streams, but the real problem is that the table is large and a fullscan is used. Just too much data to be read.

Now, let’s try the same query using CROSS APPLY / TOP … WITH TIES.

Using CROSS APPLY / TOP … WITH TIES

SELECT  COUNT(*) AS cnt, SUM(LEN(stuffing)) AS psum
FROM    (
        SELECT  DISTINCT glow
        FROM    [20091201_ties].t_distinct d
        ) dd
CROSS APPLY
        (
        SELECT  TOP 1 WITH TIES
                stuffing
        FROM    [20091201_ties].t_distinct di
        WHERE   di.glow = dd.glow
        ORDER BY
                orderer
        ) ds

View query details

This query runs for… um, 117 seconds. Thrice as slow.

Again: there are too many records. The TOP uses the index to return first rows, which is normally fast but not when there are 10,000 records to return. Using the index implies the need to lookup the values of stuffing (which we use in this query) in the actual table. You can see it in the plan above as a Nested Loops branch which joins the index (Index Seek) and the actual table (Clustered Index Seek). The overhead required for joining the index and the table by far outweighs all benefits of having preordered data.

However, this is only true for the orderers with low cardinality.

High cardinality orderers

Let’s try the same approaches but this time let’s use them to return the records holding MIN(lorderer). This field has much higher cardinality and there are few records per group that hold the MIN(lorderer) value.

Using DENSE_RANK

SELECT  COUNT(*) AS cnt, SUM(LEN(stuffing)) AS psum
FROM    (
        SELECT  d.*, DENSE_RANK() OVER (PARTITION BY glow ORDER BY lorderer) AS dr
        FROM    [20091201_ties].t_distinct d
        ) dd
WHERE   dr = 1

View query details

40 seconds.

The performance is just as poor. Since we have exactly same full table scan with the exactly same million rows to sort, it’s quite predictable.

Using CROSS APPLY / TOP … WITH TIES

SELECT  COUNT(*) AS cnt, SUM(LEN(stuffing)) AS psum
FROM    (
        SELECT  DISTINCT glow
        FROM    [20091201_ties].t_distinct d
        ) dd
CROSS APPLY
        (
        SELECT  TOP 1 WITH TIES
                stuffing
        FROM    [20091201_ties].t_distinct di
        WHERE   di.glow = dd.glow
        ORDER BY
                lorderer
        ) ds

View query details

Now, it’s completely other story. With this ordering condition, we have 10 ties per group (instead of 10,000). The index is a real benefit now, and the query completes in only 250 ms.

If we replace DISTINCT with a recursive CTE (the approach described in the previous article), the query can be improved yet a little more:

WITH    rows AS
        (
        SELECT  MIN(glow) AS glow
        FROM    [20091201_ties].t_distinct
        UNION ALL
        SELECT  glow
        FROM    (
                SELECT  di.glow, ROW_NUMBER() OVER (ORDER BY di.glow) AS rn
                FROM    rows r
                JOIN    [20091130_groupwise].t_distinct di
                ON      di.glow > r.glow
                WHERE   r.glow IS NOT NULL
                ) q
        WHERE   rn = 1
        )
SELECT  COUNT(*) AS cnt, SUM(LEN(stuffing)) AS psum
FROM    rows r
CROSS APPLY
        (
        SELECT  TOP 1 WITH TIES
                stuffing
        FROM    [20091201_ties].t_distinct di
        WHERE   di.glow = r.glow
        ORDER BY
                lorderer
        ) ds

View query details

5 ms.

Now that’s what I call performance.

Summary

To retrieve all records holding group-wise minimums or maximums (including ties), two approaches can be used in SQL Server:

  1. Use DENSE_RANK partitioned by the grouper and return the rows holding the dense rank of 1
  2. Use DISTINCT or a recursive CTE to return a list of groupers and CROSS APPLY a query with TOP 1 WITH TIES to each grouper

The first method requires sorting on (grouper, orderer), while the second one requires index scan (and implied index lookups) for each grouper.

Sorting the large amounts of data is more efficient than using an index lookup to retrieve them. However, index scan processes only the portion of data (within the tie) while sorting requires them all.

For low cardinality orderers (the values inside the MIN and MAX), too much records comprise a tie, and index scan is less efficient than a table scan with sorting. For these orderers, DENSE_RANK solution is more efficient.

For high cardinality orderers, the index scan returns only a small portion of records and it’s almost instant. For these orderers, a CROSS APPLY / TOP 1 WITH TIES is more efficient.

Hope that helps.

I’m always glad to answer the questions regarding database queries.

Ask me a question

]]> http://explainextended.com/2009/12/01/sql-server-selecting-records-holding-group-wise-maximum-with-ties/feed/ 0 SQL Server: Selecting records holding group-wise maximum http://explainextended.com/2009/11/30/sql-server-selecting-records-holding-group-wise-maximum/ http://explainextended.com/2009/11/30/sql-server-selecting-records-holding-group-wise-maximum/#comments Mon, 30 Nov 2009 20:00:11 +0000 Quassnoi http://explainextended.com/?p=3805 Continuing the series on selecting records holding group-wise maximums:

How do I select the whole records, grouped on grouper and holding a group-wise maximum (or minimum) on other column?

In this article, I’ll consider SQL Server.

SQL Server has very rich SQL syntax and its optimizer is really powerful.

However, some tricks are still required to make the queries like this to run faster.

Let’s create a sample table:

Table creation details

As in the previous articles, this table has 1,000,000 records:

  • id is the PRIMARY KEY
  • orderer is filled with random values from 1 to 10
  • glow is a low cardinality grouping field (10 distinct values)
  • ghigh is a high cardinality grouping field (10,000 distinct values)
  • stuffing is an asterisk-filled VARCHAR(200) column added to emulate payload of the actual tables

Now, let’s make a couple of queries.

Analytic functions

Just like PostgreSQL 8.4 that we considered in the previous article, SQL Server implements window (analytic) functions.

We can just copy the queries from the previous articles and see the results.

Unique rows (no ties)

SELECT  id, orderer, glow, ghigh
FROM    (
        SELECT  *, ROW_NUMBER() OVER (PARTITION BY glow ORDER BY id) AS rn
        FROM    [20091130_groupwise].t_distinct
        ) q
WHERE   rn = 1

View query details

Returning all ties

SELECT  COUNT(*) AS cnt, SUM(LEN(stuffing)) AS psum
FROM    (
        SELECT  d.*, DENSE_RANK() OVER (PARTITION BY glow ORDER BY orderer) AS dr
        FROM    [20091130_groupwise].t_distinct d
        ) dd
WHERE   dr = 1

View query details

Resolving ties by returning the MAX(id)

SELECT  id, orderer, glow, ghigh
FROM    (
        SELECT  *, ROW_NUMBER() OVER (PARTITION BY glow ORDER BY orderer, id DESC) AS rn
        FROM    [20091130_groupwise].t_distinct
        ) dd
WHERE   rn = 1

View query details

Very elegant and very slow, 9 to 38 (!) seconds.

In all cases, SQL Server uses a clustered index scan. The table is too large to fit into the cache. Many physical reads (including read-ahead reads) are required by the query.

DISTINCT / CROSS APPLY

Unlike PostgreSQL and MySQL, SQL Server does not implement any native method for returning a singe record out of a group. However, SQL Server does implement a very useful clause, CROSS APPLY, which allows taking the records from the main query and applying their results to a subquery, where they can be referenced. Unlike set-based JOIN, CROSS APPLY is record-based and clearly distinguishes between the leading and the driven tables. A driven subquery can use the values from the leading query not only in the WHERE clause but also in the TOP clause, can have its own DISTINCT and ORDER BY clauses etc.

CROSS APPLY can improve the query used in our task as well. We should just take the list of the distinct groupers and cross apply the query which would return a first record for each grouper (resolving ties as necessary).

Unique rows (no ties)

SELECT  ds.*
FROM    (
        SELECT  DISTINCT glow
        FROM    [20091130_groupwise].t_distinct d
        ) dd
CROSS APPLY
        (
        SELECT  TOP 1 id, orderer, glow, ghigh
        FROM    [20091130_groupwise].t_distinct di
        WHERE   di.glow = dd.glow
        ORDER BY
                id
        ) ds

View query details

This is quite fast, 250 ms.

Resolving ties by returning the MAX(id) within the MIN(orderer)

Since the orderer and the id are ordered differently in this query, a single CROSS APPLY would not be able to use the index efficiently:

SELECT  ds.*
FROM    (
        SELECT  DISTINCT glow
        FROM    [20091130_groupwise].t_distinct d
        ) dd
CROSS APPLY
        (
        SELECT  TOP 1 id, orderer, glow, ghigh
        FROM    [20091130_groupwise].t_distinct di
        WHERE   di.glow = dd.glow
        ORDER BY
                orderer, id DESC
        ) ds

View query details

This query has to sort to find the top record. Since there are too much records to sort, it’s very inefficient, 16 seconds.

To benefit from the index, we should use a CROSS APPLY twice: to select the MIN(orderer) and to select the MAX(id) within this orderer:

SELECT  ds.*
FROM    (
        SELECT  DISTINCT glow
        FROM    [20091130_groupwise].t_distinct d
        ) dd
CROSS APPLY
        (
        SELECT  TOP 1 dss.*
        FROM    (
                SELECT  TOP 1 orderer
                FROM    [20091130_groupwise].t_distinct di
                WHERE   di.glow = dd.glow
                ) ddd
        CROSS APPLY
                (
                SELECT  TOP 1 id, orderer, glow, ghigh
                FROM    [20091130_groupwise].t_distinct dii
                WHERE   dii.glow = dd.glow
                        AND dii.orderer = ddd.orderer
                ORDER BY
                        id DESC
                ) dss
        ) ds

View query details

Now, this query is efficient again, only 250 ms.

But can we improve the query yet a little?

Recursive CTE’s

Just like PostgreSQL 8.4, SQL Server 2005 uses recursive CTE‘s. Unfortunately, it does not use a truly set-based approach to implement them. That’s why it’s impossible to use a LIMIT / TOP clause to build the list of distinct groupers like we did in PostgreSQL 8.4.

However, SQL Server 2005 allows using window function in the recursive CTE‘s, and its optimizer is even smart enough to build an efficient plan for them that would use the TOP access path over the index.

The idea is basically the same as what we did in PostgreSQL:

  1. Find the lowest grouper in the anchor part of the recursive CTE
  2. Iteratively find the next groupers in the recursive part

The only difference is that we will have to use ROW_NUMBER instead of TOP.

This will jump over the index and find the distinct groupers in no time (of course only if there are not many of them).

Here are the queries:

Unique rows (no ties)

WITH    rows AS
        (
        SELECT  MIN(glow) AS glow
        FROM    [20091130_groupwise].t_distinct
        UNION ALL
        SELECT  glow
        FROM    (
                SELECT  di.glow, ROW_NUMBER() OVER (ORDER BY di.glow) AS rn
                FROM    rows r
                JOIN    [20091130_groupwise].t_distinct di
                ON      di.glow > r.glow
                WHERE   r.glow IS NOT NULL
                ) q
        WHERE   rn = 1
        )
SELECT  ds.*
FROM    rows r
CROSS APPLY
        (
        SELECT  TOP 1 id, orderer, glow, ghigh
        FROM    [20091130_groupwise].t_distinct d
        WHERE   d.glow = r.glow
        ORDER BY
                id DESC
        ) ds
OPTION (MAXRECURSION 0)

View query details

Resolving ties by returning the MAX(id) within the MIN(orderer)

WITH    rows AS
        (
        SELECT  MIN(glow) AS glow
        FROM    [20091130_groupwise].t_distinct
        UNION ALL
        SELECT  glow
        FROM    (
                SELECT  di.glow, ROW_NUMBER() OVER (ORDER BY di.glow) AS rn
                FROM    rows r
                JOIN    [20091130_groupwise].t_distinct di
                ON      di.glow > r.glow
                WHERE   r.glow IS NOT NULL
                ) q
        WHERE   rn = 1
        )
SELECT  ds.*
FROM    rows r
CROSS APPLY
        (
        SELECT  TOP 1 dss.*
        FROM    (
                SELECT  TOP 1 orderer
                FROM    [20091130_groupwise].t_distinct di
                WHERE   di.glow = r.glow
                ) ddd
        CROSS APPLY
                (
                SELECT  TOP 1 id, orderer, glow, ghigh
                FROM    [20091130_groupwise].t_distinct dii
                WHERE   dii.glow = r.glow
                        AND dii.orderer = ddd.orderer
                ORDER BY
                        id DESC
                ) dss
        ) ds

View query details

Both these queries are very complex and scary to look at.

However, they complete in only 3 ms. This is 3,000 (!) times as efficient as a plain ROW_NUMBER query, though the latter is much more simple to understand.

Summary

To select records holding group-wise maximums in SQL Server, the following approaches can be used:

  • Analytic functions
  • Using DISTINCT / CROSS APPLY / TOP
  • Using recursive CTE‘s / CROSS APPLY / TOP

These methods are arranged in order of increasing complexity and decreasing cost (for low-cardinality groupers).

In there are few groupers in the table, the index scan (used by DISTINCT) or a loose index scan (emulated by the recursive CTE‘s) help to build a list of distinct groupers much faster and the CROSS APPLY then uses the secondary indexes to select the first record within each group.

To be continued.

]]> http://explainextended.com/2009/11/30/sql-server-selecting-records-holding-group-wise-maximum/feed/ 0