Steve asks:

I am trying to query a log table for a web service application and determine how many concurrent sessions are in progress at each moment a transaction is executed, based on a start date and an elapsed time for each command executed through the web service. (These metrics are logged after the fact, I’m trying to write daily performance reporting for the site).

Here’s a simplified view of my base table design:

CREATE TABLE CONNECTIONS
(
USERID VARCHAR2(30),
HANDLE VARCHAR2(40),
PROCESS_DATE DATE,
COMMAND NUMBER(6,0),
COUNT NUMBER(10,0),
ELAPSED_TIME NUMBER(10,0),
NUM_OF_RECS NUMBER(10,0),
COMMAND_TIMESTAMP TIMESTAMP (6)
)

The question is: at there moment each transaction started, how many other transactions were active?

At each given moment, there is some number of active transaction. A transaction is active if the transaction begins before that moment and ends after it. This means that the moment should fall between command_timestamp and command_timestamp + elapsed_time / 86400000.

Database B-Tree indexes are not very good in queries that involve searching for a constant between two columns, so a self-join on the condition described above would be possible but not very efficient.

But these is a more simple solution.

Whenever a transaction starts, it increments the count of the open transactions. Whenever the transaction ends, it decrements it.

So we just can build a table of events: starts and ends of the transactions, ordered chronologically. Each start would be denoted with a +1, and each end with a -1. Then we should just calculate the number of the transactions open so far and subtract the number of the transactions closed.

This can be easily done merely by calculating the partial sum of these +1‘s and -1‘s, which is an easy task for Oracle‘s analytic functions.

Let’s create a sample table. I’ll put only the relevant columns there and add a stuffing column that would emulate actual payload, to measure performance:

Table creation details

BEGIN
        DBMS_RANDOM.seed(20100125);
END;
/

CREATE TABLE t_session
        (
        id NOT NULL PRIMARY KEY,
        command_timestamp NOT NULL,
        elapsed_time NOT NULL,
        stuffing NOT NULL
        )
AS
SELECT  level,
        CAST(TO_DATE('25.01.2010', 'dd.mm.yyyy') - level / 1440 - DBMS_RANDOM.value / 2880 AS TIMESTAMP(6)),
        CAST(DBMS_RANDOM.value / 360 AS NUMBER(7, 5)),
        CAST(RPAD('*', 200, '*') AS VARCHAR2(200))
FROM    dual
CONNECT BY
        level <= 1000000
/

CREATE INDEX ix_session_commandtimestamp ON t_session (command_timestamp)
/

CREATE INDEX ix_session_end ON t_session (command_timestamp + elapsed_time)
/

For the sake of brevity, elapsed_time is expressed in days, not in milliseconds.

The table is indexed with two indexes: one on command_timestamp and another one on command_timestamp + elapsed_time.

We will calculate the average of the concurrent queries for the transactions started on Jan 1st, 2010.

To do this, we will first need to select all transactions that overlap this date. This includes all transactions whose command_timestamp or command_timestamp + elapsed_time is within this date. Both these conditions are sargable, but not at the same time. An OR clause here would be inefficient, since no single index can be used to filter on both conditions.

To work around this, we will just split the query in two parts. The first part will select all transactions that started on this date, the second part will select all transactions that ended on this date but started earlier. These two sets do not intersect, and their sum gives all transaction we are interested at. So we can just merge these two sets using UNION ALL:

WITH    current_sessions AS
        (
        SELECT  *
        FROM    t_session
        WHERE   command_timestamp >= TO_DATE('01.01.2010', 'dd.mm.yyyy')
                AND command_timestamp < TO_DATE('01.01.2010', 'dd.mm.yyyy') + 1
        UNION ALL
        SELECT  *
        FROM    t_session
        WHERE   command_timestamp + elapsed_time >= TO_DATE('01.01.2010', 'dd.mm.yyyy')
                AND command_timestamp + elapsed_time < TO_DATE('01.01.2010', 'dd.mm.yyyy') + 1
                AND command_timestamp < TO_DATE('01.01.2010', 'dd.mm.yyyy')
        )
SELECT  COUNT(*), SUM(LENGTH(stuffing))
FROM    current_sessions

SELECT STATEMENT
 SORT AGGREGATE
  VIEW
   UNION-ALL PARTITION
    TABLE ACCESS BY INDEX ROWID, 20100125_concurrent.T_SESSION
     INDEX RANGE SCAN, 20100125_concurrent.IX_SESSION_COMMANDTIMESTAMP
    TABLE ACCESS BY INDEX ROWID, 20100125_concurrent.T_SESSION
     INDEX RANGE SCAN, 20100125_concurrent.IX_SESSION_END

Each part of the query used its own index and no extra effort is needed to get rid of the duplicates, so the query completes in 2 ms.

Now, we should need to calculate the number of concurrent transactions.

To do this, we will duplicate the recordset we got on the previous step, adding an extra field, event.

The first copy, with event = 1 will hold the beginnings of the transactions; the second copy with event = -1 will hold the ends. This will give us the equal number of the records with the opposite signs, so ultimately they will add up to a zero (since all transactions get into the table only after they complete). Each transaction will therefore be split into two records.

These records will then be sorted by event_date (which corresponds to command_timestamp or command_timestamp + elapsed_time respectively, depending on the set the record belongs to).

Then, the partial sum will be calculated for each record, using Oracle‘s SUM() OVER () analytic function.

Since the events are ordered by their date, the value of the partial sum will hold the number of transactions open (since their opening record had already been selected and added a +1 to the sum), but not yet closed (since their closing record had not yet been selected). We don’t know which transactions exactly contributed to the sum, but we are not interested in this information. All we know (and all we need to know) is the difference between the numbers of open and closed transactions.

On this step we have the partial sum for each record, but we need to get rid of the extra records. We are interested in the number of concurrent transactions at the moments each transaction began, so we will just filter the resultset so that it only selects the records with event = 1, since they correspond to the beginning of the transactions.

Finally, we just need to subtract 1 from the partial sums. This is because the transaction records contribute to the partial sums too, and we need to count the number of concurrent transactions, not the total transactions.

And, finally, here’s the query:

WITH    current_sessions AS
        (
        SELECT  *
        FROM    t_session
        WHERE   command_timestamp >= TO_DATE('01.01.2010', 'dd.mm.yyyy')
                AND command_timestamp < TO_DATE('01.01.2010', 'dd.mm.yyyy') + 1
        UNION ALL
        SELECT  *
        FROM    t_session
        WHERE   command_timestamp + elapsed_time >= TO_DATE('01.01.2010', 'dd.mm.yyyy')
                AND command_timestamp + elapsed_time < TO_DATE('01.01.2010', 'dd.mm.yyyy') + 1
                AND command_timestamp < TO_DATE('01.01.2010', 'dd.mm.yyyy')
        )
SELECT  AVG(concurrent), SUM(LENGTH(stuffing))
FROM    (
        SELECT  q.*,
                SUM(event) OVER (ORDER BY event_date, event DESC) - 1 AS concurrent
        FROM    (
                SELECT  1 AS event,
                        command_timestamp AS event_date,
                        cb.*
                FROM    current_sessions cb
                UNION ALL
                SELECT  -1 AS event,
                        command_timestamp + elapsed_time AS event_date,
                        ce.*
                FROM    current_sessions ce
                ) q
        )
WHERE   event = 1

SELECT STATEMENT
 TEMP TABLE TRANSFORMATION
  LOAD AS SELECT
   UNION-ALL
    TABLE ACCESS BY INDEX ROWID, 20100125_concurrent.T_SESSION
     INDEX RANGE SCAN, 20100125_concurrent.IX_SESSION_COMMANDTIMESTAMP
    TABLE ACCESS BY INDEX ROWID, 20100125_concurrent.T_SESSION
     INDEX RANGE SCAN, 20100125_concurrent.IX_SESSION_END
  SORT AGGREGATE
   VIEW
    WINDOW SORT
     VIEW
      UNION-ALL
       VIEW
        TABLE ACCESS FULL, SYS.SYS_TEMP_0FD9D6836_1825AC8
       VIEW
        TABLE ACCESS FULL, SYS.SYS_TEMP_0FD9D6836_1825AC8

The query completes in only 69 ms.

Hope that helps.

I’m always glad to answer the questions regarding database queries.

Ask me a question

There is a table that stores signal statuses that come from different devices.

• SS1 and SS2 signals are inserted to table in random times
• If either of SS1 and SS2 signal statuses is up, then the resulting signal should be up
• If both SS1 and SS2 signal statuses are down, then resulting signal should be down

I want to prepare a query that shows the result signal status changes according to SS1 and SS2 signals

Each record deals with only one signal type here: either SS1 or SS2. To obtain the signal statuses we should query the cumulative values of previous records.

If a record describes a change in SS2, we should query for the most recent change to SS1 that had been recorded so far to obtain the SS1‘s current status.

In systems other than Oracle, the previous value of a signal could be easily queried using subselects with TOP / LIMIT clauses. But Oracle does not support correlated queries nested more than one level deep, and limiting a subquery result to a single record (which is required by a subquery) requires it (ORDER BY should be nested). This makes constructing such a subquery in Oracle quite a pain.

However, in Oracle, these things can be queries using analytics functions much more efficiently.

Let’s create a sample table:

Table creation details

BEGIN
        DBMS_RANDOM.seed(20100120);
END;
/
CREATE TABLE t_signal
        (
        id NOT NULL,
        signal NOT NULL,
        status NOT NULL,
        ts NOT NULL
        )
AS
SELECT  level,
        CASE WHEN DBMS_RANDOM.value < 0.5 THEN 'SS1' ELSE 'SS2' END,
        ROUND(DBMS_RANDOM.value),
        TO_DATE('20.01.2010', 'dd.mm.yyyy') - level / 86400
FROM    dual
CONNECT BY
        level <= 1000000
/
ALTER TABLE t_signal ADD CONSTRAINT pk_signal_id PRIMARY KEY (id)
/
CREATE INDEX ix_signal_ts_id ON t_signal (ts, id)
/

This table contains 1,000,000 records filled with random states of random signals once per second.

Each record describes a state of a certain signal at a given moment of time. To find out the state of another signal at that moment of time we need to know the state held by the latest record for that signal.

To find out that state we can employ analytical function, LAST_VALUE.

With a default RANGE (that is BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW), this function just gives the value of the expression held by the current row, and, therefore, is quite useless: it gives the same result as a plain expression would give, without any functions.

If col is unique and defines the order of the rows, then LAST_VALUE(expression) OVER (ORDER BY col) is just a quite expensive synonym for expression.

However, this function’s behavior can be changed by adding IGNORE NULLS clause. This clause makes the function return the last value so far which is not a NULL.

Now, returning the cumulative value of any signal becomes quite simple. We should just just make two expressions which substitute NULL instead of the signal status for the wrong signals. The LAST_VALUE (IGNORE NULLS) over these expressions will persist until rewritten by the new states of their corresponding signals.

Let’s check it:

SELECT  *
FROM    (
        SELECT  /*+ FIRST_ROWS */
                s.*,
                DECODE(signal, 'SS1', status, NULL) AS exp1,
                DECODE(signal, 'SS2', status, NULL) AS exp2,
                LAST_VALUE(DECODE(signal, 'SS1', status, NULL) IGNORE NULLS) OVER (ORDER BY ts, id ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) AS ss1,
                LAST_VALUE(DECODE(signal, 'SS2', status, NULL) IGNORE NULLS) OVER (ORDER BY ts, id ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) AS ss2
        FROM    t_signal s
        ORDER BY
                ts, id
        ) s2
WHERE   rownum <= 15

SELECT STATEMENT
 COUNT STOPKEY
  VIEW
   WINDOW NOSORT
    TABLE ACCESS BY INDEX ROWID, 20100120_signal.T_SIGNAL
     INDEX FULL SCAN, 20100120_signal.IX_SIGNAL_TS_ID

In the resultset above, EXP1 and EXP2 show the changes in the signal states of SS1 and SS2, with NULL‘s if the current record does not describe a change in the appropriate signals.

LAST_VALUE (IGNORE NULLS) over these expressions show their cumulative values. These values can be used to calculate the resulting signal state.

Note that using ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW made Oracle to use WINDOW NOSORT and this query is instant, since the LAST_VALUE can be buffered.

To calculate the resulting signal, we should just return the GREATEST of ss1 and ss2 (returned by LAST_VALUE):

SELECT  SUM(GREATEST(ss1, ss2))
FROM    (
        SELECT  s.*,
                DECODE(signal, 'SS1', status, NULL) AS exp1,
                DECODE(signal, 'SS2', status, NULL) AS exp2,
                LAST_VALUE(DECODE(signal, 'SS1', status, NULL) IGNORE NULLS) OVER (ORDER BY ts, id ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) AS ss1,
                LAST_VALUE(DECODE(signal, 'SS2', status, NULL) IGNORE NULLS) OVER (ORDER BY ts, id ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) AS ss2
        FROM    t_signal s
        ORDER BY
                ts, id
        ) s2

SELECT STATEMENT
 SORT AGGREGATE
  VIEW
   WINDOW SORT
    TABLE ACCESS FULL, 20100120_signal.T_SIGNAL

For the sake of brevity, we select the SUM of the resulting signal states which gives us the number of records with resulting signal up. As it should be, this number roughly amounts to 75% of the total number of records.

The query uses a single sort (which in case of the whole table is faster than traversing the index), buffers the results and completes in 3 seconds which is almost as fast as a plain query with ORDER BY over the same dataset would complete.

I am making an inner join of two tables.

I have many time intervals represented by intervals and i want to get measure data from measures only within those intervals. The intervals do not overlap.

Here are the table layouts:

intervals

entry_time	exit_time

measures

time	measure

There are 1295 records in intervals and about a million records in measures. The intervals do not overlap.

The result I want to get is a table with in the first column the measure, then the time the measure has been done, the begin/end time of the considered interval (it would be repeated for row with a time within the considered range)

How can I make this faster?

Straight query

The query to get the results would look like this:

SELECT  measures.measure as measure,
        measures.time as time,
        intervals.entry_time as entry_time,
        intervals.exit_time as exit_time
FROM    intervals
JOIN    measures
ON      measures.time BETWEEN intervals.entry_time AND intervals.exit_time
ORDER BY
        time ASC

This looks quite simple. However, choosing the execution plan can be somewhat tricky task for this query.

The table layout gives us no hint on how many rows will be returned from the join. If all intervals begin before the time of the first measure and end after the time of the last measure, then every combination of rows will be returned from both tables and the resulting recordset will in fact be a cartesian product of two tables. If the intervals are short and sparse, few or even no rows can be returned from measures.

However, we know that the intervals do not overlap. This means that each measure may belong to at most one interval and the number of records in the final resultset will be no more than the number of records in measures.

Since the condition we join here is a pair of inequalities, only two methods can be used by the engine to perform the join, that is NESTED LOOPS and MERGE JOIN. HASH JOIN which is most efficient on large tables requires an equijoin and cannot be used for this query.

MERGE JOIN sorts both resultsets on the columns it joins on and gradually advances the internal pointer on both sorted row sources so that the values of the columns being joined on in both tables always match. In case of an equijoin, the engine returns only the records holding the current value of the pointer; in case of a join on inequality condition, the engine returns all records greater (or less) than the current value from the corresponding table.

MERGE JOIN, however, can only satisfy a single inequality condition while we here have two of them on two different columns. The engine should split this condition into the pair of inequalities: the first one will be used by the join; the second one will be just filtered. The query essentially turns into this one:

SELECT  measures.measure as measure,
        measures.time as time,
        intervals.entry_time as entry_time,
        intervals.exit_time as exit_time
FROM    intervals
JOIN    measures
ON      measures.time >= intervals.entry_time
WHERE   measures.time  <= intervals.exit_time
ORDER BY
        time ASC

Here, the MERGE JOIN will be using the predicate in the ON clause and the filtering will use the predicate in the WHERE clause. The predicates are symmetrical and can easily be swapped.

However, the join will have to return all records to the filtering code. And with an inequality condition like the one we see above, there will be lots of records. If we take a normal situation: the interval bounds are more or less in accordance with the dates of the first and last measure and the intervals are distributed evenly, then the MERGE JOIN will return (1265 × 1,000,000) / 2 ≈ 600,000,000 records, each to be filtered on the next step.

From performance’s point of view, this is hardly different from a cartesian join: in fact, it needs to process only as few as a half of the rows. These rows, however, need to be sorted beforehand (or taken from the index which is slow to traverse in its sort order), so this can actually even be slower than a cartesian join which does not require the rows to be in any certain order and can use an INDEX FAST FULL SCAN or a TABLE SCAN.

This means the only efficient way to run this query is using NESTED LOOPS. To benefit from this method, we should make an index on measures.time and convince Oracle to use it.

Let’s create a sample table:

Table creation details

BEGIN
        DBMS_RANDOM.seed(20091228);
END;
/

CREATE TABLE intervals (
        entry_time NOT NULL,
        exit_time NOT NULL
)
AS
SELECT  TO_DATE('25.12.2009', 'dd.mm.yyyy') - end - len,
        TO_DATE('25.12.2009', 'dd.mm.yyyy') - end
FROM    (
        SELECT  SUM(s) OVER (ORDER BY lvl ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) AS end,
                s * l AS len
        FROM    (
                SELECT  level AS lvl,
                        DBMS_RANDOM.value * 10 AS s, DBMS_RANDOM.value AS l
                FROM    dual
                CONNECT BY
                        level <= 1500
                )
        )
/

CREATE UNIQUE INDEX ux_intervals_entry ON intervals (entry_time)
/

CREATE UNIQUE INDEX ux_intervals_exit ON intervals (exit_time)
/

CREATE TABLE measures (
        time NOT NULL,
        measure NOT NULL,
        stuffing NOT NULL
)
AS
SELECT  TO_DATE('28.12.2009', 'dd.mm.yyyy') - level / 144,
        CAST(DBMS_RANDOM.value * 10000 AS NUMBER(18, 2)),
        CAST(RPAD('*', 200, '*') AS VARCHAR2(200))
FROM    dual
CONNECT BY
        level <= 1080000
/

ALTER TABLE measures ADD CONSTRAINT pk_measures_time PRIMARY KEY (time)
/

CREATE INDEX ix_measures_time_measure ON measures (time, measure)
/

There are 1,500 non-overlapping intervals with random lengths from 1 to 10 days, and 1,080,000 measures. The overall spans of the intervals and the measures roughly match. entry_time and exit_time in intervals are indexed with UNIQUE indexes, and there is a composite index on measures (time, measure). measures also contains a stuffing column which is a VARCHAR2(200) filled with asterisks. This emulates actual tables containing lots of data.

Here’s a query that selects all measures within the corresponding intervals and finds the total length of their stuffing columns:

SELECT  /*+ ORDERED USE_NL (i, m) */
        SUM(LENGTH(stuffing))
FROM    intervals i
JOIN    measures m
ON      m.time BETWEEN entry_time AND exit_time

SELECT STATEMENT
 SORT AGGREGATE
  TABLE ACCESS BY INDEX ROWID, 20091228_intervals.MEASURES
   NESTED LOOPS
    TABLE ACCESS FULL, 20091228_intervals.INTERVALS
    INDEX RANGE SCAN, 20091228_intervals.PK_MEASURES_TIME

This takes a little more than 10 seconds.

The same query using MERGE JOIN ran for more than 10 minutes on my test machine and I had to interrupt it after that.

So the NESTED LOOPS is most efficient algorithm that Oracle can use for this query and it completes in 10.35 seconds.

Interval hashing

NESTED LOOPS algorithm is quite efficient for a query like this, but requires traversing the index in a loop which results in a TABLE ACCESS BY INDEX ROWID that is needed to find the values of stuffing. In this very case a HASH JOIN with a FULL TABLE SCAN would be more efficient.

HASH JOIN requires an equality condition so we need to provide it somehow. This can be done by using a technique called interval hashing.

Here’s how it works:

Splitting the time axis

The time axis is mapped into a set of ranges each being assigned with an ordinal number. The mapping function applied to a timestamp should uniquely define the range it belongs to. Usually this is done by splitting the time axis to the ranges of equal length and taking the integer part of the difference between the timestamp and the beginning of the first range divided by the range length:

Here’s an expression to do this:

TRUNC((timestamp - TO_DATE(1, 'J')) * 2)

This expression splits the time axis into a set ranges each being 12 hours long.

Finding the ranges

For each interval, all ranges it overlaps should be found:

This can be easily done by mapping the beginning and the end of each interval to their corresponding ranges and finding all the ranges between them.

This query returns the first range, the last range and total number of ranges each interval overlaps:

SELECT  entry_range, exit_range,
        exit_range - entry_range + 1 AS range_span,
        entry_time, exit_time
FROM    (
        SELECT  TRUNC((entry_time - TO_DATE(1, 'J')) * 2) AS entry_range,
                TRUNC((exit_time - TO_DATE(1, 'J')) * 2) AS exit_range,
                entry_time,
                exit_time
        FROM    intervals
        )

Exploding the ranges

We should explode the ranges: if an interval spans n ranges, we should generate a recordset of n records for this interval, each corresponding to an individual range.

This would be an easy task if Oracle did support generate_series (like PostgreSQL) or CROSS APPLY (like SQL Server). Unfortunately, it does support neither of these, so we will have to make do with a simple join.

We should do the following:

• Find the longest interval in terms of ranges it overlaps:

  

  On the picture above, the longest interval spans 3 ranges.

  Here’s a query to find the interval that spans most ranges:

  WITH    splits AS
          (
          SELECT  /*+ MATERIALIZE */
                  entry_range, exit_range,
                  exit_range - entry_range + 1 AS range_span,
                  entry_time, exit_time
          FROM    (
                  SELECT  TRUNC((entry_time - TO_DATE(1, 'J')) * 2) AS entry_range,
                          TRUNC((exit_time - TO_DATE(1, 'J')) * 2) AS exit_range,
                          entry_time,
                          exit_time
                  FROM    intervals
                  )
          ),
          upper AS
          (
          SELECT  /*+ MATERIALIZE */
                  MAX(range_span) AS max_range
          FROM    splits
          )
  SELECT  *
  FROM    upper
  

  I added the MATERIALIZE hint so that the CTE results are stored in the temporary tablespace and not reevaluated each time the CTE is called. This will be used later.

• Generate the dummy rowset of as many records as there are the ranges the longest interval overlaps:

  WITH    splits AS
          (
          SELECT  /*+ MATERIALIZE */
                  entry_range, exit_range,
                  exit_range - entry_range + 1 AS range_span,
                  entry_time, exit_time
          FROM    (
                  SELECT  TRUNC((entry_time - TO_DATE(1, 'J')) * 2) AS entry_range,
                          TRUNC((exit_time - TO_DATE(1, 'J')) * 2) AS exit_range,
                          entry_time,
                          exit_time
                  FROM    intervals
                  )
          ),
          upper AS
          (
          SELECT  /*+ MATERIALIZE */
                  MAX(range_span) AS max_range
          FROM    splits
          ),
          ranges AS
          (
          SELECT  /*+ MATERIALIZE */
                  level AS chunk
          FROM    upper
          CONNECT BY
                  level <= max_range
          )
  SELECT  *
  FROM    ranges
  

• Join the intervals to this rowset, generating one record per each range an interval overlaps:

  

  This is done using this query:

  WITH    splits AS
          (
          SELECT  /*+ MATERIALIZE */
                  entry_range, exit_range,
                  exit_range - entry_range + 1 AS range_span,
                  entry_time, exit_time
          FROM    (
                  SELECT  TRUNC((entry_time - TO_DATE(1, 'J')) * 2) AS entry_range,
                          TRUNC((exit_time - TO_DATE(1, 'J')) * 2) AS exit_range,
                          entry_time,
                          exit_time
                  FROM    intervals
                  )
          ),
          upper AS
          (
          SELECT  /*+ MATERIALIZE */
                  MAX(range_span) AS max_range
          FROM    splits
          ),
          ranges AS
          (
          SELECT  /*+ MATERIALIZE */
                  level AS chunk
          FROM    upper
          CONNECT BY
                  level <= max_range
          ),
          tiles AS
          (
          SELECT  /*+ MATERIALIZE USE_MERGE (r s) */
                  entry_range + chunk - 1 AS tile,
                  entry_time,
                  exit_time
          FROM    ranges r
          JOIN    splits s
          ON      chunk <= range_span
          )
  SELECT  *
  FROM    tiles
  

  Note that I added the hint USE_MERGE to the query that does the actual join. Since this is a pure inequality condition, without extra filtering, a MERGE JOIN is actually a best plan here.

  This query returns one record for each time an interval overlaps a range. Not that though the intervals themselves do not overlap, two intervals can overlap the same range and it will be returned twice in this case.

• Now we have a recordset with the range numbers.

Joining on the ranges

When we have the intervals mapped into the ranges we can do just the same with the measures: map the timestamps into the ranges and join on them.

Since this would be an equijoin, a HASH JOIN method can be used:

View query details

WITH    splits AS
        (
        SELECT  /*+ MATERIALIZE */
                entry_range, exit_range,
                exit_range - entry_range + 1 AS range_span,
                entry_time, exit_time
        FROM    (
                SELECT  TRUNC((entry_time - TO_DATE(1, 'J')) * 2) AS entry_range,
                        TRUNC((exit_time - TO_DATE(1, 'J')) * 2) AS exit_range,
                        entry_time,
                        exit_time
                FROM    intervals
                )
        ),
        upper AS
        (
        SELECT  /*+ MATERIALIZE */
                MAX(range_span) AS max_range
        FROM    splits
        ),
        ranges AS
        (
        SELECT  /*+ MATERIALIZE */
                level AS chunk
        FROM    upper
        CONNECT BY
                level <= max_range
        ),
        tiles AS
        (
        SELECT  /*+ MATERIALIZE USE_MERGE (r s) */
                entry_range + chunk - 1 AS tile,
                entry_time,
                exit_time
        FROM    ranges r
        JOIN    splits s
        ON      chunk <= range_span
        )
SELECT  /*+ LEADING(t) USE_HASH(m t) */
        SUM(LENGTH(stuffing))
FROM    tiles t
JOIN    measures m
ON      TRUNC((m.time - TO_DATE(1, 'J')) * 2) = tile

SELECT STATEMENT
 TEMP TABLE TRANSFORMATION
  LOAD AS SELECT , 20091228_intervals.MEASURES
   TABLE ACCESS FULL, 20091228_intervals.INTERVALS
  LOAD AS SELECT , 20091228_intervals.MEASURES
   SORT AGGREGATE
    VIEW
     TABLE ACCESS FULL, SYS.SYS_TEMP_0FD9D67D5_1825AC8
  LOAD AS SELECT , 20091228_intervals.MEASURES
   CONNECT BY WITHOUT FILTERING
    COUNT
     VIEW
      TABLE ACCESS FULL, SYS.SYS_TEMP_0FD9D67D6_1825AC8
  LOAD AS SELECT , 20091228_intervals.MEASURES
   MERGE JOIN
    SORT JOIN
     VIEW
      TABLE ACCESS FULL, SYS.SYS_TEMP_0FD9D67D7_1825AC8
    SORT JOIN
     VIEW
      TABLE ACCESS FULL, SYS.SYS_TEMP_0FD9D67D5_1825AC8
  SORT AGGREGATE
   HASH JOIN
    VIEW
     TABLE ACCESS FULL, SYS.SYS_TEMP_0FD9D67D8_1825AC8
    TABLE ACCESS FULL, 20091228_intervals.MEASURES

We see that this query is almost 5 times as fast. Unfortunately, it returns incorrect results.

This is because joining on the ranges is not enough. The fact that both an interval and a measure overlap each other means they overlap the same range (and hence the match will be satisfied by the join). But the opposite is not true: even if an interval and a measure overlap the same range, they do not necessarily overlap each other.

Adding fine filtering

This problem however, can be fixed by adding a simple condition to the WHERE clause of the query: it will be exactly the same condition for checking the intervals bounds against the measure time that we originally used.

Since the query returns everything we need (plus some excess data we don’t need), we just need to filter the incorrect matches out.

This is essentially what the original query did, but instead of making N×M comparisons required by the cartesian join or half that number required by the MERGE JOIN, the comparisons will only need to be made inside a single range. If we divide the time axis into R ranges, then R ranges should be examined, with (N/R)×(M/R) comparisons within each range. This means that the total number of comparisons will be (N×M)/R which of course is much more efficient.

Here’s the final query:

WITH    splits AS
        (
        SELECT  /*+ MATERIALIZE */
                entry_range, exit_range,
                exit_range - entry_range + 1 AS range_span,
                entry_time, exit_time
        FROM    (
                SELECT  TRUNC((entry_time - TO_DATE(1, 'J')) * 2) AS entry_range,
                        TRUNC((exit_time - TO_DATE(1, 'J')) * 2) AS exit_range,
                        entry_time,
                        exit_time
                FROM    intervals
                )
        ),
        upper AS
        (
        SELECT  /*+ MATERIALIZE */
                MAX(range_span) AS max_range
        FROM    splits
        ),
        ranges AS
        (
        SELECT  /*+ MATERIALIZE */
                level AS chunk
        FROM    upper
        CONNECT BY
                level <= max_range
        ),
        tiles AS
        (
        SELECT  /*+ MATERIALIZE USE_MERGE (r s) */
                entry_range + chunk - 1 AS tile,
                entry_time,
                exit_time
        FROM    ranges r
        JOIN    splits s
        ON      chunk <= range_span
        )
SELECT  /*+ LEADING(t) USE_HASH(m t) */
        SUM(LENGTH(stuffing))
FROM    tiles t
JOIN    measures m
ON      TRUNC((m.time - TO_DATE(1, 'J')) * 2) = tile
        AND m.time BETWEEN t.entry_time AND t.exit_time

SELECT STATEMENT
 TEMP TABLE TRANSFORMATION
  LOAD AS SELECT , 20091228_intervals.MEASURES
   TABLE ACCESS FULL, 20091228_intervals.INTERVALS
  LOAD AS SELECT , 20091228_intervals.MEASURES
   SORT AGGREGATE
    VIEW
     TABLE ACCESS FULL, SYS.SYS_TEMP_0FD9D67DD_1825AC8
  LOAD AS SELECT , 20091228_intervals.MEASURES
   CONNECT BY WITHOUT FILTERING
    COUNT
     VIEW
      TABLE ACCESS FULL, SYS.SYS_TEMP_0FD9D67DE_1825AC8
  LOAD AS SELECT , 20091228_intervals.MEASURES
   MERGE JOIN
    SORT JOIN
     VIEW
      TABLE ACCESS FULL, SYS.SYS_TEMP_0FD9D67DF_1825AC8
    SORT JOIN
     VIEW
      TABLE ACCESS FULL, SYS.SYS_TEMP_0FD9D67DD_1825AC8
  SORT AGGREGATE
   HASH JOIN
    VIEW
     TABLE ACCESS FULL, SYS.SYS_TEMP_0FD9D67E0_1825AC8
    TABLE ACCESS FULL, 20091228_intervals.MEASURES

This query completes in only 2.53 seconds.

Sorry for not posting and answering for so long, had a few urgent things to do, and hope these breaks won’t be so long in the future :)

Now, continuing the series on selecting records holding group-wise maximums:

How do I select the whole records, grouped on grouper and holding a group-wise maximum (or minimum) on other column?

Finally, Oracle.

We will create the very same table as we did for the previous systems:

Table creation details

CREATE TABLE t_distinct
        (
        id NOT NULL,
        orderer NOT NULL,
        glow NOT NULL,
        ghigh NOT NULL,
        stuffing NOT NULL
        )
AS
SELECT  level,
        TRUNC(DBMS_RANDOM.value * 10) + 1,
        MOD(level - 1, 10) + 1,
        MOD(level - 1, 10000) + 1,
        CAST(RPAD('*', 200, '*') AS VARCHAR2(200))
FROM    dual
CONNECT BY
        level <= 1000000
/
ALTER TABLE t_distinct ADD CONSTRAINT pk_distinct_id PRIMARY KEY (id)
/
CREATE INDEX ix_distinct_glow_id ON t_distinct (glow, id)
/
CREATE INDEX ix_distinct_ghigh_id ON t_distinct (ghigh, id)
/
CREATE INDEX ix_distinct_glow_orderer_id ON t_distinct (glow, orderer, id)
/
CREATE INDEX ix_distinct_ghigh_orderer_id ON t_distinct (ghigh, orderer, id)
/

There are 1,000,000 records with the following fields:

• id is the PRIMARY KEY
• orderer is filled with random values from 1 to 10
• glow is a low cardinality grouping field (10 distinct values)
• ghigh is a high cardinality grouping field (10,000 distinct values)
• stuffing is an asterisk-filled VARCHAR(200) column added to emulate payload of the actual tables

Analytic functions

Of course Oracle support analytic functions just like SQL Server and PostgreSQL (and actually, that’s SQL Server and PostgreSQL that support analytics functions like Oracle).

To select records holding the group-wise maximums, we can apply the very same ROW_NUMBER and DENSE_RANK solutions we used for SQL Server and PostgreSQL before:

SELECT  id, orderer, glow, ghigh
FROM    (
        SELECT  d.*, ROW_NUMBER() OVER (PARTITION BY glow ORDER BY id) AS rn
        FROM    t_distinct d
        ) q
WHERE   rn = 1

View query details

SELECT STATEMENT
 VIEW
  WINDOW SORT PUSHED RANK
   TABLE ACCESS FULL, 20091216_groupwise.T_DISTINCT

SELECT  COUNT(*) AS cnt, SUM(LENGTH(stuffing)) AS psum
FROM    (
        SELECT  d.*, DENSE_RANK() OVER (PARTITION BY glow ORDER BY orderer) AS dr
        FROM    t_distinct d
        ) dd
WHERE   dr = 1

View query details

SELECT STATEMENT
 SORT AGGREGATE
  VIEW
   WINDOW SORT PUSHED RANK
    TABLE ACCESS FULL, 20091216_groupwise.T_DISTINCT

This takes a little more than 6 seconds.

CONNECT BY and subqueries

Unfortunately in Oracle we cannot use the decisions we used to improve the query time in the other systems.

Oracle does not support loose index scan on composite indexes like MySQL.

It does support a similar solution, INDEX SKIP SCAN, but can use it only for the queries that search for the exact value of the orderer, not for MIN or MAX.

Oracle does not support CROSS APPLY so we cannot apply a multirecord or a multicolumn query to each result of the rowset.

More than that: Oracle does not event support deep correlation of the subqueries. You cannot nest a reference to the outer table in the subquery, and using ROW_NUMBER or ROWNUM to filter the first result requires nesting. So making an analog of MySQL‘s LIMIT in a subquery won’t work either.

Oracle 10g (which is what I’m writing about here) also does not support recursive CTE‘s, so it’s not possible to use this method too.

However, there still is a way to improve the query.

Instead of jumping over the index values to build the list of distinct groupers, we can build a whole list of the possible groupers and join it with the original table.

The possible groupers are or course those between MIN and MAX.

To build such a list we can use a CONNECT BY query over the dual table, (which is a standard way to generate random rowsets in Oracle), and just take MIN(glow) and MAX(glow) as the start and stop conditions.

We then can find the MIN(orderer) or MIN(id) for each grouper from the generated list and join the table back on this value. This will use an efficient INDEX RANGE SCAN (MIN / MAX). There can be gaps: not all groupers will be present in the list, but this is not a problem, since the subquery will return a NULL and the join will just yield no rows.

Here’s the query to find the records holding the MIN(id):

WITH    q AS
        (
        SELECT  level - 1 +
                (
                SELECT  MIN(glow)
                FROM    t_distinct
                ) AS d
        FROM    dual
        CONNECT BY
                level <=
                (
                SELECT  MAX(glow)
                FROM    t_distinct
                ) -
                (
                SELECT  MIN(glow)
                FROM    t_distinct
                ) + 1
        )
SELECT  d.id, d.orderer, d.glow, d.glow
FROM    q
JOIN    t_distinct d
ON      d.id =
        (
        SELECT  MAX(id)
        FROM    t_distinct di
        WHERE   di.glow = q.d
        )

SELECT STATEMENT
 NESTED LOOPS
  VIEW
   CONNECT BY WITHOUT FILTERING
    FAST DUAL
  TABLE ACCESS BY INDEX ROWID, 20091216_groupwise.T_DISTINCT
   INDEX UNIQUE SCAN, 20091216_groupwise.PK_DISTINCT_ID
    SORT AGGREGATE
     FIRST ROW
      INDEX RANGE SCAN (MIN/MAX), 20091216_groupwise.IX_DISTINCT_GLOW_ID

, and this is a query to find the records holding MAX(id) within the MIN(orderer):

WITH    q AS
        (
        SELECT  level - 1 +
                (
                SELECT  MIN(glow)
                FROM    t_distinct
                ) AS d
        FROM    dual
        CONNECT BY
                level <=
                (
                SELECT  MAX(glow)
                FROM    t_distinct
                ) -
                (
                SELECT  MIN(glow)
                FROM    t_distinct
                ) + 1
        )
SELECT  d.id, d.orderer, d.glow, d.glow
FROM    (
        SELECT  d,
                (
                SELECT  MIN(orderer)
                FROM    t_distinct d1
                WHERE   d1.glow = q.d
                ) AS mo
        FROM    q
        ) q2
JOIN    t_distinct d
ON      d.id =
        (
        SELECT  MAX(id)
        FROM    t_distinct d2
        WHERE   d2.glow = q2.d
                AND d2.orderer = q2.mo
        )

SELECT STATEMENT
 NESTED LOOPS
  VIEW
   VIEW
    CONNECT BY WITHOUT FILTERING
     FAST DUAL
  TABLE ACCESS BY INDEX ROWID, 20091216_groupwise.T_DISTINCT
   INDEX UNIQUE SCAN, 20091216_groupwise.PK_DISTINCT_ID
    SORT AGGREGATE
     FIRST ROW
      INDEX RANGE SCAN (MIN/MAX), 20091216_groupwise.IX_DISTINCT_GLOW_ORDERER_ID

The latter query is a little bit more complex since we need two subqueries here: one to find the MIN(orderer), another one to find the MAX(id). The principle is the same, however.

Both subqueries are very fast and complete within 1 millisecond (read instantly).

This query depends not only on the selectivity of the grouper but also on the actual values of the bounds (the higher is the difference between MIN and MAX on the groupers, the less efficient the query is).

For this solution to work, the groupers should have a well-ordered datatype. This means you can build a list of all possible values between MIN and MAX. Integers and dates (without time portion) are well-ordered; real numbers, strings and timestamps are not.

For this solution to be efficient, the groupers should be well-ordered, dense and have low cardinality. Though these limitations can be quite strong, the solutions is still extremely useful for very many many real-world scenarios which groups by integers (like categories in the blogging system or department codes in an accounting application); or dates without time portion (like sales reports).

Suppose I have this table

A	B	C	D

Datatypes are not important.

I want to do this:

SELECT  a AS a1, b AS b1, c AS c1,
        (
        SELECT  SUM(d)
        FROM    source
        WHERE   a = a1
                AND b = b1
        ) AS total
FROM    source
GROUP BY
        a, b, c

, but I can’t find a way (SQL Developer keeps complaining with FROM clause not found.)

Is there a way? Is it possible?

This is of course possible if we just alias the query and prepend the alias to the field:

SELECT  a, b, c,
        (
        SELECT  SUM(d)
        FROM    source si
        WHERE   si.a = so.a
                AND si.b = so.b
        CONNECT BY
                16 >= level
        )
FROM    source so
GROUP BY
        a, b, c

This works well on this sample set of data:

WITH    source AS
        (
        SELECT  FLOOR(MOD(level - 1, 8) / 4) + 1 AS a,
                FLOOR(MOD(level - 1, 4) / 2) + 1 AS b,
                MOD(level - 1, 2) + 1 AS c,
                level AS d
        FROM    dual
        CONNECT BY
                16 >= level
        )
SELECT  a, b, c,
        (
        SELECT  SUM(d)
        FROM    source si
        WHERE   si.a = so.a
                AND si.b = so.b
        )
FROM    source so
GROUP BY
        a, b, c

 
View query details

, but it needs to reevaluate the subquery for each group.

In Oracle, there is a better way to do this query: nesting an aggregate SUM inside an analytical SUM.

What this query does is in fact the following:

• Calculate the groupwise SUM(d) for each (a, b)
• Retrieve the distinct values of a, b, c
• For each (a, b, c), return the SUM(d) for the corresponding (a, b)

This means that total will be the same for each (a, b), thous the values of c may differ.

However, the SUM is associative, that is sum of groupwise sums is the same as the sum of separate values.

To calculate the SUM(d) for a given (a, b) we can calculate the partial sums for (a, b, c) and add them together. Then we need return this value for each (a, b) (which can be multiple).

To return an aggregate value along with each of the records that contribute to it, the analytical functions are used (those with OVER clause).

It is widely known that analytical functions, as well as aggregate functions, cannot be nested.

However, one can nest an aggregate function within an analytical one. This is possible because aggregate functions work over the source-level values before GROUP BY, while analytical ones work with SELECT-level expressions (after the GROUP BY)

Here’s the query to do this:

WITH    source AS
        (
        SELECT  FLOOR(MOD(level - 1, 8) / 4) + 1 AS a,
                FLOOR(MOD(level - 1, 4) / 2) + 1 AS b,
                MOD(level - 1, 2) + 1 AS c,
                level AS d
        FROM    dual
        CONNECT BY
                level <= 16
        )
SELECT  a, b, c, SUM(SUM(d)) OVER (PARTITION BY a, b)
FROM    source
GROUP BY
        a, b, c

View query details

SELECT STATEMENT
 WINDOW BUFFER
  SORT GROUP BY
   VIEW
    CONNECT BY WITHOUT FILTERING
     FAST DUAL

This query returns the same results.

The innermost SUMs are calculated (a, b, c)-wise (like in GROUP BY), while the outermost ones are calculated (a, b)-wise (like in PARTITION BY).

Since ordering by (a, b, c) implies ordering by (a, b), both these groupings are performed with a single SORT.

Jason asks:

I have a co-worker who swears that Oracle IN queries are slow and refuses to use them. For example:

SELECT  foo
FROM    bar
WHERE   bar.stuff IN
        (
        SELECT  stuff
        FROM    asdf
        )

Typically, this kind of performance advice strikes me as overgeneralized and my instinct is to ignore it. But I figure I’ll give him the benefit of the doubt and ask about it.

So, in general is an IN query very expensive? I’m having trouble putting together many non-trivial queries to run on EXPLAIN PLAN.

Does the IN predicate always have inferior efficiency compared to it’s counterparts, EXISTS and JOIN?

Let’s check.

First of all, there are at least three ways to check each row in a table against a list of (possible duplicate) values and return each row at most once:

• IN predicate:

  SELECT  foo
  FROM    bar
  WHERE   bar.stuff IN
          (
          SELECT  stuff
          FROM    asdf
          )
  

• EXISTS predicate:

  SELECT  foo
  FROM    bar
  WHERE   EXISTS
          (
          SELECT  NULL
          FROM    asdf
          WHERE   asdf.stuff = bar.stuff
          )
  

• JOIN / DISTINCT:

  SELECT  b.foo
  FROM    (
          SELECT  DISTINCT stuff
          FROM    asdf
          ) a
  JOIN    bar b
  ON      b.stuff = a.stuff
  

All these queries are semantically the same.

Common wisdom advices against using IN predicates because it just doesn’t look a right thing to do.

This is because less experienced SQL developers tend to translate the SQL statements into pseudocode just like they see it, which in this case looks something like this:

foreach ($bar as $bar_record) {
    foreach ($asdf as $asdf_record) {
        if ($bar_record->stuff == $asdf_record->stuff)
            output ($bar_record);
            break;
    }
}

This just radiates inefficiency, since the inner rowset should be iterated for each row returned.

EXISTS looks more nice since it at least gives a hint of possibility to use an index. However, it still looks like the same nested loops.

Finally, the JOIN looks the most promising here since everybody knows joins are optimized (though few know how exactly). But an IN? No thanks, they say, it will execute once for every row, it’s slow, it’s bad, it’s inefficient!

People, come on. Oracle developers have thought it over ages ago and guess what: they turned out to be smart enough to implement an efficient algorithm for an IN construct.

Now, let’s make two sample tables and see how it’s done:

Table creation details

CREATE TABLE t_outer (id NOT NULL, value NOT NULL, stuffing NOT NULL)
AS
SELECT  level, level, CAST(RPAD('*', 100, '*') AS VARCHAR2(100))
FROM    dual
CONNECT BY
        level <= 10
/

CREATE TABLE t_inner (id NOT NULL, pvalue NOT NULL, uvalue NOT NULL, stuffing NOT NULL)
AS
SELECT  level, FLOOR((level - 1) / 2) + 1, level, CAST(RPAD('*', 100, '*') AS VARCHAR2(100))
FROM    dual
CONNECT BY
        level <= 1000000
/

ALTER TABLE t_outer ADD CONSTRAINT pk_outer_id PRIMARY KEY (id)
/
ALTER TABLE t_inner ADD CONSTRAINT pk_inner_id PRIMARY KEY (id)
/
CREATE UNIQUE INDEX ux_inner_uvalue ON t_inner (uvalue)
/

Table t_outer contains 10 records.

Table t_inner contains 1,000,000 records. The field uvalue has a UNIQUE index defined on it, while the field pvalue has no index at all, and contains duplicates (each value from 1 to 500,000 is stored twice).

Now, let’s make a couple of queries.

IN on UNIQUE column

SELECT  id, value
FROM    t_outer
WHERE   value IN
        (
        SELECT  uvalue
        FROM    t_inner
        )

View query details

SELECT STATEMENT
 NESTED LOOPS
  TABLE ACCESS FULL, 20090930_in.T_OUTER
  INDEX UNIQUE SCAN, 20090930_in.UX_INNER_UVALUE

This query returns all 10 values from the t_outer instantly.

If we look into the query plan we will see that this is just a plain NESTED LOOPS join on the index. No whole subquery reevaluation, the index is used and used efficiently.

Oracle is smart enough to make three logical constructs:

1. IN is equivalent to a JOIN / DISTINCT
2. DISTINCT on a column marked as UNIQUE and NOT NULL is redundant, so the IN is equivalent to a simple JOIN
3. IN is equivalent to a simple JOIN so any valid join method and the access methods can be used

This makes this query super fast.

JOIN / DISTINCT on UNIQUE column

SELECT  id, value
FROM    t_outer o
JOIN    (
        SELECT  DISTINCT uvalue
        FROM    t_inner
        )
ON      o.value = uvalue

View query details

SELECT STATEMENT
 VIEW
  NESTED LOOPS
   TABLE ACCESS FULL, 20090930_in.T_OUTER
   INDEX UNIQUE SCAN, 20090930_in.UX_INNER_UVALUE

Exactly same query plan. From Oracle‘s point of view, these queries are identical. Oracle just sees that they are the same and can avoid the whole query evaluation at all: it just uses index access.

Now let’s see how smart Oracle will be when it will have to search for a non-indexed column which, in addition, contains duplicates.

JOIN / DISTINCT on a non-indexed column

SELECT  id, value
FROM    t_outer o
JOIN    (
        SELECT  DISTINCT pvalue
        FROM    t_inner
        )
ON      o.value = pvalue

View query details

SELECT STATEMENT
 VIEW
  HASH UNIQUE
   HASH JOIN
    TABLE ACCESS FULL, 20090930_in.T_OUTER
    TABLE ACCESS FULL, 20090930_in.T_INNER

Now the query takes quite a significant time, more than 2 seconds.

Oracle decided to make the join first and then get rid of the duplicates, despite the fact that the DISTINCT clause is inside the inline view, not outside. However, semantically this is the same and Oracle knows it.

Now, how will the IN predicate work? Will it be so smart to convert the query to the JOIN which completes in 2 seconds or it will be running the inner query over and over and over again?

Let’s see.

IN on a non-indexed column

SELECT  id, value
FROM    t_outer
WHERE   value IN
        (
        SELECT  pvalue
        FROM    t_inner
        )

View query details

SELECT STATEMENT
 HASH JOIN SEMI
  TABLE ACCESS FULL, 20090930_in.T_OUTER
  TABLE ACCESS FULL, 20090930_in.T_INNER

1 millisecond.

The plan says TABLE ACCESS FULL on both tables, and the t_inner is very large, there are 1,000,000 rows there and they are long rows.

It just cannot be scanned in 1 ms. However, this is what we see. How can it be?

If we look closer to the plan we will see a special method there, a HASH JOIN SEMI. This is a such called semi-join. It is designed just for the queries like this, using IN or EXISTS with an equality condition on the correlated field.

This method works as follows:

• It builds a hash table on the outer query (the one that is probed). I repeat: it is the outer query that is hashed, not the inner one. Everything that needs to be returned in SELECT clause goes to the hash table
• Then it takes every row from the inner table and searches the hash table for the value of the column being tested
• If the value is found, the engine returns all records from the hash table that match this value and deletes them from the hash table. Unlike the plain JOIN, the hash table is dynamic: it is populated before the JOIN begins and each row is removed from it as its gets returned
• When either the hash table or the subquery is out if records, the query completes

Instead of removing the duplicates from the subquery before the iteration begins, the engine removes the values from the hash table as long as the iteration goes. This cannot be easily rewritten in SQL since SQL operates on immutable sets. However, outside of the set theory this can be implemented easily.

The duplicates in the subquery are not a problem anymore. Imagine we have value 42 three times in the outer table and five times in the inner table. The hash table will be build and all three outer query records will get into this hash table.

Now, the first value of 42 is fetched out of the subquery. This will result in a hash hit, all three records sharing this value will be found, returned and, which is most important, deleted.

What happens when the second value of 42 will be fetched? Well, nothing will happen. There are no more records in the hash table, they were deleted as they were returned. That’s why this duplicate will just result in a hash miss and go away. Same is true for all other duplicates.

Now we can see what happened in our query. The engine build the hash table over the 10 values from t_outer. Then it took the first value from t_inner (which happened to be a 1), found it in the hash table and returned the hashed record from t_outer. The second value of 1 fetched from the subquery resulted in a hash miss and was promptly forgotten. In 20 records the hash table was empty. And when the hash table is empty, there is no point in running the query further: nothing could possibly be returned anymore.

Of course, there was a little cheating here: first 10 numbers just happened to be fetched out in first 10 records. If we would search for values from 499,990 to 500,000, the query would run for the same 2 seconds. However, in average, the hash table would be emptied much faster than the DISTINCT set could be built to use with a JOIN.

EXISTS

Oracle treats EXISTS predicate with an equality condition for a correlated value exactly the same as an IN predicate. Let’s see it:

EXISTS on unique value

SELECT  id, value
FROM    t_outer
WHERE   EXISTS
        (
        SELECT  NULL
        FROM    t_inner
        WHERE   uvalue = value
        )

SELECT STATEMENT
 NESTED LOOPS SEMI
  TABLE ACCESS FULL, 20090930_in.T_OUTER
  INDEX UNIQUE SCAN, 20090930_in.UX_INNER_UVALUE

EXISTS on a non-unique value

SELECT  id, value
FROM    t_outer
WHERE   EXISTS
        (
        SELECT  NULL
        FROM    t_inner
        WHERE   pvalue = value
        )

SELECT STATEMENT
 HASH JOIN SEMI
  TABLE ACCESS FULL, 20090930_in.T_OUTER
  TABLE ACCESS FULL, 20090930_in.T_INNER

Exactly same plans, results and performance.

Summary

IN predicate is an ugly duckling of SQL. Due to the fact it appears inefficient in the eye of an inexperienced developer, it is generally considered non-efficient and EXISTS or a JOIN are advised to use instead.

However, Oracle optimizes IN fairly well, using indexes, UNIQUE constraints, NOT NULL modifiers and other extra information wisely.

Even in case of duplicates and absence of an index on the subquery field, Oracle is able to use HASH SEMI JOIN method which is more efficient than a JOIN / DISTINCT solution.

EXISTS is optimized exactly the same, however, IN is more readable and concise.

IN predicate is exactly what should be use to search records against a list of values in Oracle.

Hope that helps.

I’m always glad to answer the questions regarding database queries.

Ask me a question

What is better to store hierarchical data: nested sets model or adjacency list (parent-child) model?

For detailed explanations of the terms, see the first article in the series:

• Adjacency list vs. nested sets: PostgreSQL

Today is Oracle time.

Adjacent sets require recursion, and Oracle has a native way to implement recursion in SQL statements. To do this it uses a special clause, CONNECT BY.

Unlike other systems, Oracle always supported recursion in SQL statements since version 2 (this is the name the first version of Oracle was released under).

This was done because by that time there already were several hierarchical databases on the market which are quite good for storing hierarchical data. If not for this clause, transition from a hierarchical DBMS to the relational DBMS would be too hard.

Oracle‘s way to query self-relations is somewhat different from recursive CTE‘s that other systems use:

• Recursive CTEs can use an arbitrary set in recursive steps. The recursion base and recursion stack are not visible in further steps. The recursive operation is a set operation (usually a JOIN) on the recursion parameter (a set) and the result of the previous operation:

  

• CONNECT BY queries can only use the value of a single row in recursive steps.

  The recursion base and recursion stack are visible, the values outside the recursion stack are not.

  The recursive operation is a set operation on each row of the current row and the base rowset.

  Unlike recursive CTE, CONNECT BY operates row-wise, that is each row of the result spans its own recursion branch with its own stack:

  

The main difference it that the CONNECT BY operation cannot produce anything that was not in the original rowset, while a recursive CTE can.

However, CONNECT BY has numerous benefits too: it’s very fast, allows to track the hierarchy path easily (using a built-in funtion named SYS_CONNECT_BY_PATH) and implicitly sorts the recordset in tree order (with additional clause ORDER SIBLINGS BY to sort the siblings).

This method, of course, allows traversing adjacency lists easily.

To compare efficiency of adjacency lists and that of nested sets, let’s create a sample table:

Table creation details

CREATE TABLE t_hierarchy (id NOT NULL, parent NOT NULL, lft NOT NULL, rgt NOT NULL, data NOT NULL, stuffing NOT NULL)
AS
WITH    nums AS
        (
        SELECT  level AS num, 100 AS value
        FROM    dual
        CONNECT BY
                level <= 5
        ),
        digits AS
        (
        SELECT  SYS_CONNECT_BY_PATH(num, '/') AS digit, level AS len
        FROM    nums
        CONNECT BY
                level <= 9
        )
SELECT  id,
        FLOOR((id - 1) / 5),
        lft,
        lft + FLOOR(2929687 / POWER(5, len)) - 1,
        CAST('Value ' || id AS VARCHAR2(100)),
        CAST(RPAD('*', 100, '*') AS VARCHAR2(100))
FROM    (
        SELECT  digit, len,
                (
                SELECT  SUM(SUBSTR(digit, level * 2, 1) * POWER(5, len - level))
                FROM    dual
                CONNECT BY
                        level <= len
                ) AS id,
                (
                SELECT  SUM((SUBSTR(digit, level * 2, 1) - 1) * FLOOR(2929687 / POWER(5, level)) + 1)
                FROM    dual
                CONNECT BY
                        level <= len
                ) AS lft
        FROM    digits
        )
/
ALTER TABLE t_hierarchy ADD CONSTRAINT pk_hierarchy_id PRIMARY KEY (id)
/
CREATE INDEX ix_hierarchy_parent ON t_hierarchy (parent)
/
CREATE INDEX ix_hierarchy_lft ON t_hierarchy (lft)
/
CREATE INDEX ix_hierarchy_rgt ON t_hierarchy (rgt)
/

The table contains both adjacency list data and nested sets data, with 8 levels of nesting, 5 children of each parent node and 2,441,405 records.

As in the previous articles, we’ll test performance of the three most used queries:

• Find all descendants of a given node
• Find all ancestors of a given node
• Find all descendants of a given node up to a certain depth

All descendants

Nested sets

SELECT  SUM(LENGTH(hc.stuffing))
FROM    t_hierarchy hp
JOIN    t_hierarchy hc
ON      hc.lft BETWEEN hp.lft AND hp.rgt
WHERE   hp.id = 42

View query details

SELECT STATEMENT
 SORT AGGREGATE
  NESTED LOOPS
   TABLE ACCESS BY INDEX ROWID, 20090928_nested.T_HIERARCHY
    INDEX UNIQUE SCAN, 20090928_nested.PK_HIERARCHY_ID
   TABLE ACCESS BY INDEX ROWID, 20090928_nested.T_HIERARCHY
    INDEX RANGE SCAN, 20090928_nested.IX_HIERARCHY_LFT

This is super fast, faster than in any system before that: only 10 ms. The plan is quite predictable: a UNIQUE scan to find the parent record and a range scan to find all children.

Adjacency list

SELECT  SUM(LENGTH(stuffing))
FROM    t_hierarchy
START WITH
        id = 42
CONNECT BY
        parent = PRIOR id

View query details

SELECT STATEMENT
 SORT AGGREGATE
  CONNECT BY WITH FILTERING
   TABLE ACCESS BY INDEX ROWID, 20090928_nested.T_HIERARCHY
    INDEX UNIQUE SCAN, 20090928_nested.PK_HIERARCHY_ID
   NESTED LOOPS
    BUFFER SORT
     CONNECT BY PUMP
    TABLE ACCESS BY INDEX ROWID, 20090928_nested.T_HIERARCHY
     INDEX RANGE SCAN, 20090928_nested.IX_HIERARCHY_PARENT
   TABLE ACCESS FULL, 20090928_nested.T_HIERARCHY

This is quite fast too: 140 ms. The plan loops over the parent ranges for each id.

Note the TABLE ACCESS FULL in the end of the plan. Oracle reserves a right to do a FULL SCAN instead of a RANGE SCAN if it considers a certain parent range too large to use the index. However, this is recursion, not a single operation, and it’s impossible to tell in advance which id values will be returned on each recursion step. That’s why Oracle shows both methods, just in case, and chooses the most efficient method in runtime.

In our table, at most 5 rows share a parent (of more then 2 million), so it’s very unlikely this plan branch will be ever chosen.

Find all ancestors

Nested sets

SELECT  hp.id, hp.parent, hp.lft, hp.rgt, hp.data
FROM    t_hierarchy hc
JOIN    t_hierarchy hp
ON      hc.lft BETWEEN hp.lft AND hp.rgt
WHERE   hc.id = 1000000
ORDER BY
        hp.lft

View query details

SELECT STATEMENT
 NESTED LOOPS
  TABLE ACCESS BY INDEX ROWID, 20090928_nested.T_HIERARCHY
   INDEX UNIQUE SCAN, 20090928_nested.PK_HIERARCHY_ID
  TABLE ACCESS BY INDEX ROWID, 20090928_nested.T_HIERARCHY
   INDEX RANGE SCAN, 20090928_nested.IX_HIERARCHY_LFT

The plan used is exactly the same as for the query to choose all descendants.

However, the range on lft is too broad and the query works for more than 3 seconds.

Adjacency list

SELECT  id, parent, lft, rgt, data, level
FROM    t_hierarchy
START WITH
        id = 1000000
CONNECT BY
        id = PRIOR parent
ORDER BY
        level DESC

View query details

SELECT STATEMENT
 SORT ORDER BY
  CONNECT BY WITH FILTERING
   TABLE ACCESS BY INDEX ROWID, 20090928_nested.T_HIERARCHY
    INDEX UNIQUE SCAN, 20090928_nested.PK_HIERARCHY_ID
   NESTED LOOPS
    BUFFER SORT
     CONNECT BY PUMP
    TABLE ACCESS BY INDEX ROWID, 20090928_nested.T_HIERARCHY
     INDEX UNIQUE SCAN, 20090928_nested.PK_HIERARCHY_ID
   TABLE ACCESS FULL, 20090928_nested.T_HIERARCHY

This query completes in less than 1 ms, which is instant.

Note that unlike recursive CTE‘s, Oracle supplies a built-in preudocolumn, level, that returns the current recursion level. This comes handy in cases like this.

Descendants up to a given level

Nested sets

SELECT  hc.id, hc.parent, hc.lft, hc.rgt, hc.data
FROM    t_hierarchy hp
JOIN    t_hierarchy hc
ON      hc.lft BETWEEN hp.lft AND hp.rgt
WHERE   hp.id = ?
        AND
        (
        SELECT  COUNT(*)
        FROM    t_hierarchy hn
        WHERE   hc.lft BETWEEN hn.lft AND hn.rgt
                AND hn.lft BETWEEN hp.lft AND hp.rgt
        ) <= 3

View the query for node 42

SELECT  hc.id, hc.parent, hc.lft, hc.rgt, hc.data
FROM    t_hierarchy hp
JOIN    t_hierarchy hc
ON      hc.lft BETWEEN hp.lft AND hp.rgt
WHERE   hp.id = 42
        AND
        (
        SELECT  COUNT(*)
        FROM    t_hierarchy hn
        WHERE   hc.lft BETWEEN hn.lft AND hn.rgt
                AND hn.lft BETWEEN hp.lft AND hp.rgt
        ) <= 3

SELECT STATEMENT
 NESTED LOOPS
  TABLE ACCESS BY INDEX ROWID, 20090928_nested.T_HIERARCHY
   INDEX UNIQUE SCAN, 20090928_nested.PK_HIERARCHY_ID
  TABLE ACCESS BY INDEX ROWID, 20090928_nested.T_HIERARCHY
   INDEX RANGE SCAN, 20090928_nested.IX_HIERARCHY_LFT
    SORT AGGREGATE
     FILTER
      TABLE ACCESS BY INDEX ROWID, 20090928_nested.T_HIERARCHY
       INDEX RANGE SCAN, 20090928_nested.IX_HIERARCHY_LFT

View the query for node 31,415

SELECT  hc.id, hc.parent, hc.lft, hc.rgt, hc.data
FROM    t_hierarchy hp
JOIN    t_hierarchy hc
ON      hc.lft BETWEEN hp.lft AND hp.rgt
WHERE   hp.id = 31415
        AND
        (
        SELECT  COUNT(*)
        FROM    t_hierarchy hn
        WHERE   hc.lft BETWEEN hn.lft AND hn.rgt
                AND hn.lft BETWEEN hp.lft AND hp.rgt
        ) <= 3

SELECT STATEMENT
 NESTED LOOPS
  TABLE ACCESS BY INDEX ROWID, 20090928_nested.T_HIERARCHY
   INDEX UNIQUE SCAN, 20090928_nested.PK_HIERARCHY_ID
  TABLE ACCESS BY INDEX ROWID, 20090928_nested.T_HIERARCHY
   INDEX RANGE SCAN, 20090928_nested.IX_HIERARCHY_LFT
    SORT AGGREGATE
     FILTER
      TABLE ACCESS BY INDEX ROWID, 20090928_nested.T_HIERARCHY
       INDEX RANGE SCAN, 20090928_nested.IX_HIERARCHY_LFT

As in other systems, performance heavily depends on how many descendants does the record have.

For the node 42 which is close to the root node and has lots of descendants, the query runs for 122 seconds. For the node 31,415, the query takes 1.5 ms (next to instant).

Adjacency list

SELECT  id, parent, lft, rgt, data
FROM    t_hierarchy
START WITH
        id = ?
CONNECT BY
        parent = PRIOR id
        AND level <= 3

Limiting the recursion depth query is very simple in Oracle: we just add the additional condition to CONNECT BY clause constraining the limit.

View the query for node 42

SELECT  id, parent, lft, rgt, data
FROM    t_hierarchy
START WITH
        id = 42
CONNECT BY
        parent = PRIOR id
        AND level <= 3

SELECT STATEMENT
 CONNECT BY WITH FILTERING
  TABLE ACCESS BY INDEX ROWID, 20090928_nested.T_HIERARCHY
   INDEX UNIQUE SCAN, 20090928_nested.PK_HIERARCHY_ID
  NESTED LOOPS
   BUFFER SORT
    CONNECT BY PUMP
   FILTER
    TABLE ACCESS BY INDEX ROWID, 20090928_nested.T_HIERARCHY
     INDEX RANGE SCAN, 20090928_nested.IX_HIERARCHY_PARENT
  TABLE ACCESS FULL, 20090928_nested.T_HIERARCHY

View the query for node 31,415

SELECT  id, parent, lft, rgt, data
FROM    t_hierarchy
START WITH
        id = 31415
CONNECT BY
        parent = PRIOR id
        AND level <= 3

SELECT STATEMENT
 CONNECT BY WITH FILTERING
  TABLE ACCESS BY INDEX ROWID, 20090928_nested.T_HIERARCHY
   INDEX UNIQUE SCAN, 20090928_nested.PK_HIERARCHY_ID
  NESTED LOOPS
   BUFFER SORT
    CONNECT BY PUMP
   FILTER
    TABLE ACCESS BY INDEX ROWID, 20090928_nested.T_HIERARCHY
     INDEX RANGE SCAN, 20090928_nested.IX_HIERARCHY_PARENT
  TABLE ACCESS FULL, 20090928_nested.T_HIERARCHY

Both queries complete in 1 ms, that is instant.

Summary

Oracle implements a special clause, CONNECT BY, that makes traversing adjacency lists in general and parent-child relationships in particular very easy, since this is exactly what this clause was intended for.

Except for finding all descendants of a single node (without finding out or limiting the depth), nested sets model is less efficient in Oracle than adjacency lists.

Adjacency lists are not only more efficient but the queries using them are more elegant and easy to maintain due to the native support.

That’s why adjacency list model should be used instead of nested sets model in Oracle too, as well as in SQL Server and PostgreSQL 8.4.

Cora asks:

I want to build an alphabetical index over the articles in my database.

I need to show 5 entries in the index, like this: A-D, E-F etc., so that each entry contains roughly equal number of articles.

How do I do it in a single query?

This is in Oracle

This is a good task to demonstrate Oracle‘s analytic abilities.

To do this, we will create a sample table:

Table creation details

BEGIN
        DBMS_RANDOM.seed(20090923);
END;
/
CREATE TABLE t_name (id NOT NULL PRIMARY KEY, name NOT NULL)
AS
SELECT  level, CAST(SUBSTR('ETAOINSHRDLUCMFWYPVBGKQJXZ', 1 + FLOOR(LOG(0.85, POWER(0.85, 26) + DBMS_RANDOM.value() * (1 - POWER(0.85, 26)))), 1) || ' article ' || level AS VARCHAR(50)) AS name
FROM    dual
CONNECT BY
        level <= 10000
/
CREATE INDEX ix_name_name ON t_name (name)
/

This script creates 10,000 random articles with the names distributed logarithmically, according to ETAOIN SHRDLU.

Here’s the distribution:

View the distribution of initial letters

SELECT  SUBSTR(name, 1, 1) AS letter, COUNT(*) AS cnt
FROM    t_name
GROUP BY
        SUBSTR(name, 1, 1)
ORDER BY
        cnt DESC

To split the letters into 5 tomes (contiguous letter sets), we should calculate the aggregated sum of the articles starting with each letters, divide it by overall sum of them articles and distribute the letters according to this value.

This can be done with Oracle‘s ability to calculate partial sums and groupwise sums without actual grouping.

To calculate partial sums, we should use SUM(cnt) OVER (ORDER BY letter). This will calculate and accumulate the count of articles starting from each letter.

To calculate total sum and assign it to each row, we just use SUM(cnt) OVER(), without ORDER BY clause. This is an equivalent of an aggregate SUM, but is returned for each row.

Here are the results:

View the partial and total sums and their ratios

SELECT  letter, cnt,
        SUM(cnt) OVER (ORDER BY letter) AS partial_sum,
        SUM(cnt) OVER () AS total_sum,
        TO_CHAR(SUM(cnt) OVER (ORDER BY letter) / SUM(cnt) OVER (), 'FM0.990') AS partial_ratio
FROM    (
        SELECT  SUBSTR(name, 1, 1) AS letter, COUNT(*) AS cnt
        FROM    t_name
        GROUP BY
                SUBSTR(name, 1, 1)
        )

Now we should just take each value and assign it to the closest quotient of 5: 0.0, 0.2, 0.4 etc.

However, there is a problem.

The query above returns the following results for D and E:

0.0 means tome 1, 0.2 means tome 2.

0.2 is inside the E range which is from 0.180 to 0.338.

Should we assign E to tome 1 or tome 2?

Assigning E to tome 1 makes it 0.338 articles long or 0.138 articles longer than it should be, while assigning to tome 2 makes it only 0.020 articles shorter.

It seems that the second option is more fair, since the difference is 0.020 vs. 0.138 in the first case.

So to calculate the tome the letter should go to, we should minimize the difference.

To do this, we should take the average of the previous and the current value of ratio (in this case, it would be (0.338 + 0.180) / 2 = 0.259) and round it down to the nearest quotient (0.2).

To calculate the difference between the rows, one could use LAG function. However, the analytic function cannot be nested and this would require nesting the whole query.

But there is a more elegant way.

Remember the algorithm the ratios are calculated with? They are the partial sums divided by the total sum.

The average of two consecutive ratios is the sum of all values but the current one plus the sum of all values (including the current one) divided by two and divided by the total sum:

fair_ratio = AVG(partial_ratio, previous_partial_ratio) = SUM(1:N-1) + SUM(1:N) / 2 * SUM()

But the sum of all values including the current one is in fact the sum of all preceding values plus the current value.

So to get the average of the ratios, we can just take the sum of all preceding values excluding the current one and add a half of the current value to it.

And Oracle allows us to calculate the sum of all values but the current one merely by adding the RANGE clause to the analytical function. This expression:

SUM(cnt) OVER (ORDER BY letter RANGE BETWEEN UNBOUNDED PRECEDING AND 1 PRECEDING) + cnt / 2

returns the average of the current and the previous partial sums by calculating the previous partial sum and adding half the current value to it.

Let’s use it in a query:

View the query with fair ratios

SELECT  letter, cnt,
        SUM(cnt) OVER (ORDER BY letter) AS partial_sum,
        SUM(cnt) OVER () AS total_sum,
        TO_CHAR(SUM(cnt) OVER (ORDER BY letter) / SUM(cnt) OVER (), 'FM0.990') AS partial_ratio,
        TO_CHAR((COALESCE(SUM(cnt) OVER (ORDER BY letter ROWS BETWEEN UNBOUNDED PRECEDING AND 1 PRECEDING), 0) + cnt / 2) / SUM(cnt) OVER (), 'FM0.990') AS fair_ratio
FROM    (
        SELECT  SUBSTR(name, 1, 1) AS letter, COUNT(*) AS cnt
        FROM    t_name
        GROUP BY
                SUBSTR(name, 1, 1)
        )

We see that for letter E the fair_ratio is 0.259, which is the average of the partial_ratios from the preceding and the current rows:

Now we cal easily distribute the letters between the tomes.

To do this, we should just multiply the fair_ratio by 5 and round it down to the closest integer using FLOOR. This will give us the tome number.

The first and the last letters of each tome can be easily calculated using MIN and MAX.

Here’s the query to do this:

SELECT  tome, MIN(letter) || '-' || MAX(letter) AS range, SUM(cnt)
FROM    (
        SELECT  letter, cnt,
                TRUNC((COALESCE(SUM(cnt) OVER (ORDER BY letter ROWS BETWEEN UNBOUNDED PRECEDING AND 1 PRECEDING), 0) + cnt / 2) / SUM(cnt) OVER () * 5) + 1 AS tome
        FROM    (
                SELECT  SUBSTR(name, 1, 1) AS letter, COUNT(*) AS cnt
                FROM    t_name
                GROUP BY
                        SUBSTR(name, 1, 1)
                )
        )
GROUP BY
        tome
ORDER BY
        tome

As you can see, the distribution is reasonably fair.

Hope that helps.

I’m always glad to answer the questions regarding database queries.

Ask me a question

Which method is best to select values present in one table but missing in another one?

This:

SELECT  l.*
FROM    t_left l
LEFT JOIN
        t_right r
ON      r.value = l.value
WHERE   r.value IS NULL

, this:

SELECT  l.*
FROM    t_left l
WHERE   l.value NOT IN
        (
        SELECT  value
        FROM    t_right r
        )

or this:

SELECT  l.*
FROM    t_left l
WHERE   NOT EXISTS
        (
        SELECT  NULL
        FROM    t_right r
        WHERE   r.value = l.value
        )

Today, we will see how Oracle copes with these queries.

And to do this, we, of course, should create sample tables:

Table creation script

CREATE TABLE t_left (id NOT NULL PRIMARY KEY, value NOT NULL, stuffing NOT NULL)
AS
SELECT  level, MOD(level, 10000), RPAD('Value ' || level || ' ', 200, '*')
FROM    dual
CONNECT BY
        level <= 100000
/

CREATE TABLE t_right (id NOT NULL PRIMARY KEY, value NOT NULL, stuffing NOT NULL) AS
SELECT  level, MOD(level, 10000) + 1, CAST(RPAD('Value ' || level || ' ', 200, '*') AS VARCHAR2(200))
FROM    dual
CONNECT BY
        level <= 1000000
/

CREATE INDEX ix_left_value ON t_left (value)
/

CREATE INDEX ix_right_value ON t_right (value)
/

Table t_left contains 100,000 rows with 10,000 distinct values.

Table t_right contains 1,000,000 rows with 10,000 distinct values.

There are 10 rows in t_left with values not present in t_right.

In both tables the field value is indexed.

LEFT JOIN / IS NULL

SELECT  l.value, l.id
FROM    t_left l
LEFT JOIN
        t_right r
ON      r.value = l.value
WHERE   r.value IS NULL

View query results and execution plan

SELECT STATEMENT
 HASH JOIN ANTI
  VIEW , 20090917_anti.index$_join$_001
   HASH JOIN
    INDEX FAST FULL SCAN, 20090917_anti.SYS_C0013641
    INDEX FAST FULL SCAN, 20090917_anti.IX_LEFT_VALUE
  INDEX FAST FULL SCAN, 20090917_anti.IX_RIGHT_VALUE

This execution plan is quite interesting.

First, Oracle‘s optimizer, unlike SQL Server‘s one, is smart enough to see an opportunity to use ANTI JOIN for such a query.

Since all rows from t_left should be examined, Oracle decided to use a HASH ANTI JOIN to do this: a hash table is built over the values from t_right, eliminating duplicates, and every row from t_left is searched for in the hash table. Rows not found in the hash table are returned; those that are found are not.

Second, Oracle decided to optimize the query yet a little.

In the table we actually have three fields: id (a PRIMARY KEY), value we are searching for, and stuffing. The latter field is a VARCHAR(200) and filled with asterisks. This field emulates values in actual tables that makes the tables large in size.

Since the tables in Oracle are heap organized by default, no index contains values of both id and value.

However, there are two indexes that contain them separately: the index on value created explicitly, and the index on id created implicitly to police the PRIMARY KEY.

This indexes, even taken together, are still much smaller than the table itself. And that’s why Oracle decided to retrieve the values not from the table but from the indexes joined on rowid.

Indeed, each of the index records corresponds to a record in the table. And index record contains a rowid: a pointer to the original record in the table. Having equal rowid‘s means belonging to the same row. And rowid is of course unique in each index, since one table record corresponds to exactly one index record.

Since we don’t use other columns except id and value, these two indexes contain all information we need. And joining them, though takes some extra time, is still faster than scanning the whole table (which as we know is quite large and occupies a lot of data blocks).

The HASH JOIN we see in the plan does the things described above. Here are the steps it performs:

1. First, it scans one of the two indexes with FAST FULL SCAN. This scan methods is faster since it does not follow the B-Tree links but instead just reads all index pages sequentially. This does not preserve the index order but we don’t need it anyway.
2. Then it builds a hash table from the index records read on step 1 (using ROWID‘s of the records as a hash condition)
3. Finally, it reads the second index (again, with a FAST FULL SCAN) and probes it against the hash table

This results in the same rowset of (id, value) we would obtain had we scanned the table, only the HASH JOIN method is faster.

The query completes in 0.27 s.

NOT IN

SELECT  l.id, l.value
FROM    t_left l
WHERE   value NOT IN
        (
        SELECT  value
        FROM    t_right
        )

View query results and execution plan

SELECT STATEMENT
 HASH JOIN ANTI
  VIEW , 20090917_anti.index$_join$_001
   HASH JOIN
    INDEX FAST FULL SCAN, 20090917_anti.PK_LEFT_ID
    INDEX FAST FULL SCAN, 20090917_anti.IX_LEFT_VALUE
  INDEX FAST FULL SCAN, 20090917_anti.IX_RIGHT_VALUE

NOT IN is exactly what ANTI JOIN is designed for, that’s why it’s no surprise to see that Oracle uses ANTI JOIN for NOT IN.

NOT IN is semantically different from the LEFT JOIN / IS NULL and NOT EXISTS since its logic is trivalent and filtering on it will never return anything if there are NULL values in the list.

This behaviour was explained in the first article of the series (that concerning SQL Server) and this is a reason why PostgreSQL is reluctant to use an Anti Join for NOT IN.

However, Oracle is able to take into account the fact that t_right.value is declared as NOT NULL, and, therefore, NOT IN is semantically equivalent to LEFT JOIN / IS NULL and NOT EXISTS. And Oracle uses exactly same plan for NOT IN, with an ANTI JOIN and a HASH JOIN to get (id, value) for t_left.

The query completes in 0.28 s, same as for LEFT JOIN / IS NULL.

NOT EXISTS

SELECT  l.id, l.value
FROM    t_left l
WHERE   NOT EXISTS
        (
        SELECT  value
        FROM    t_right r
        WHERE   r.value = l.value
        )

View query results and execution plan

SELECT STATEMENT
 HASH JOIN ANTI
  VIEW , 20090917_anti.index$_join$_001
   HASH JOIN
    INDEX FAST FULL SCAN, 20090917_anti.PK_LEFT_ID
    INDEX FAST FULL SCAN, 20090917_anti.IX_LEFT_VALUE
  INDEX FAST FULL SCAN, 20090917_anti.IX_RIGHT_VALUE

Guess what?

Same execution plan, same results, same performance.

The query completes in 0.28 s.

Summary

Oracle‘s optimizer is able to see that NOT EXISTS, NOT IN and LEFT JOIN / IS NULL are semantically equivalent as long as the list values are declared as NOT NULL.

It uses same execution plan for all three methods, and they yield same results in same time.

In Oracle, it is safe to use any method of the three described above to select values from a table that are missing in another table.

However, if the values are not guaranteed to be NOT NULL, LEFT JOIN / IS NULL or NOT EXISTS should be used rather than NOT IN, since the latter will produce different results depending on whether or not there are NULL values in the subquery resultset.

I have a table with data such as below:

Group	Start Date	End Date
A	2001.01.01	2001.01.03
A	2001.01.01	2001.01.02
A	2001.01.03	2001.01.03
B	2001.01.01	2001.01.01

I am looking to produce a view that gives a count for each day

We have a list of ranges here, and for each date we should count the number of ranges that contain this date.

To make this query, we will employ one simple fact: the number of ranges containing a given date is the number of ranges started on or before this date minus the number of ranges that ended before this date.

This can easily be calculated using window functions.

Let’s create a sample table:

Table creation details

BEGIN
        DBMS_RANDOM.seed(20090909);
END;
/

CREATE TABLE t_range (
        id NOT NULL PRIMARY KEY,
        grouper NOT NULL,
        start_date NOT NULL,
        end_date NOT NULL
)
AS
SELECT  level, MOD((level - 1), 10) + 1,
        TO_DATE('09.09.2009', 'dd.mm.yyyy') - TRUNC((level - 1) / 14400),
        TO_DATE('09.09.2009', 'dd.mm.yyyy') - TRUNC((level - 1) / 14400) +
        TRUNC(DBMS_RANDOM.value * 10) + 1
FROM    dual
CONNECT BY
        level <= 1000000
/
CREATE INDEX ix_range_grouper_startdate ON t_range (grouper, start_date)
/
CREATE INDEX ix_range_grouper_enddate ON t_range (grouper, end_date)
/

This table contains 1,000,000 rows in 10 groups.

Ranges start every 10 minutes (144 per day) and last from 1 to 11 days. Range lengths are random.

To generate the count of ranges in each group that contain each date, we need to do the following:

1. Generate a list of dates, from minimal to maximal. This can be easily done using a CONNECT BY query on dual table:

   SELECT  cur_date
   FROM    (
           SELECT  (
                   SELECT  MIN(start_date)
                   FROM    t_range
                   ) + level - 1 AS cur_date
           FROM    dual
           CONNECT BY
                   level <=
                   (
                   SELECT  MAX(end_date)
                   FROM    t_range
                   ) -
                   (
                   SELECT  MIN(start_date)
                   FROM    t_range
                   ) + 1
           ) dates
   

   CUR_DATE
   02.07.2009 00:00:00
   03.07.2009 00:00:00
   04.07.2009 00:00:00
   18.09.2009 00:00:00
   19.09.2009 00:00:00
   80 rows fetched in 0.0017s (0.0011s)

   Since MIN and MAX are instant on indexed fields, this query is instant too.

2. Build a rowset that would contain a number of ranges starting on each given date. We can use a simple GROUP BY to do this, since missing dates will be handled by a LEFT JOIN with the list of generated dates that we built on the previous step:

   SELECT  grouper AS sgrp, start_date, COUNT(*) AS scnt
   FROM    t_range
   GROUP BY
           grouper, start_date
   ORDER BY
           grouper, start_date
   

   SGRP	START_DATE	SCNT
   1	02.07.2009 00:00:00	640
   1	03.07.2009 00:00:00	1440
   1	04.07.2009 00:00:00	1440
   10	08.09.2009 00:00:00	1440
   10	09.09.2009 00:00:00	1440
   700 rows fetched in 0.0247s (0.3159s)

   We see that within group 1, 640 ranges started on 02.07.2009, 1,400 ranges started on 03.07.2009 etc.

3. Do the same with end_dates:

   SELECT  grouper AS egrp, end_date, COUNT(*) AS ecnt
   FROM    t_range
   GROUP BY
           grouper, end_date
   ORDER BY
           grouper, end_date
   

   EGRP	END_DATE	ECNT
   1	03.07.2009 00:00:00	56
   1	04.07.2009 00:00:00	197
   1	05.07.2009 00:00:00	359
   10	18.09.2009 00:00:00	287
   10	19.09.2009 00:00:00	139
   790 rows fetched in 0.0278s (0.6250s)

   Within group 1, 56 ranges ended on 03.07.2009, 197 ones ended on 04.07.2009 etc.

4. Finally, we should calculate the number of ranges within each group containing each date:

   1. Before and on Jul 2nd, 640 ranges started. 0 ranges ended before that date. This means that this day is contained by 640 ranges.

   2. Before and on Jul 3rd, 2,080 ranges started. This includes 1,400 ranges that started on that date and 640 ranges that started before. No ranges ended before this date too. The date is contained by 2,080 ranges.

   3. Before and on Jul 4th, 3,520 ranges started. This includes 1,400 ranges that started on that date and 2,080 ranges that started before that date. 56 ranges ended before that date. This date is therefore contained by 3,462 ranges.

   Et cetera. Now we see that to calculate the number of ranges contained by any given date we should OUTER JOIN the resultsets containing the counts, and calculate partial sums of these counts. Difference between these sums will be the result we’re after.

And here’s the final query:

SELECT  cur_date,
        grouper,
        SUM(COALESCE(scnt, 0) - COALESCE(ecnt, 0)) OVER (PARTITION BY grouper ORDER BY cur_date) AS ranges
FROM    (
        SELECT  (
                SELECT  MIN(start_date)
                FROM    t_range
                ) + level - 1 AS cur_date
        FROM    dual
        CONNECT BY
                level <=
                (
                SELECT  MAX(end_date)
                FROM    t_range
                ) -
                (
                SELECT  MIN(start_date)
                FROM    t_range
                ) + 1
        ) dates
CROSS JOIN
        (
        SELECT  DISTINCT grouper AS grouper
        FROM    t_range
        ) groups
LEFT JOIN
        (
        SELECT  grouper AS sgrp, start_date, COUNT(*) AS scnt
        FROM    t_range
        GROUP BY
                grouper, start_date
        ) starts
ON      sgrp = grouper
        AND start_date = cur_date
LEFT JOIN
        (
        SELECT  grouper AS egrp, end_date, COUNT(*) AS ecnt
        FROM    t_range
        GROUP BY
                grouper, end_date
        ) ends
ON      egrp = grouper
        AND end_date = cur_date - 1
ORDER BY
        grouper, cur_date

The results are just like we expected.

Note the JOIN condition end_date = cur_date - 1. We subtract a day to make sure that we select events that ended strictly before the date. The events that ended on the date in question still contain it and should contribute into the count.

This query is very fast (1,000,000 rows are processed in just a trifle more than a second).