New Sample: Query_FillMissingValues

by AD Tejpal

This sample db demonstrates filling up of missing values in a sequential series. Three types of source data are covered:
(a) Number series.
(b) Alpha-numeric series.
(c) Date series.

Subform on left shows the start and end values for each missing range of sequential values per PersonID. Subform at right depicts all the missing sequential values duly filled in. Each record representing start of a new sequence is highlighted in color.

Note:
A single field table (T_Ref) populated with sequential numbers from 0 onwards is used as the driver for eliciting the missing values. The number of records in this table should be such as to cover the maximum likely range of missing values.

Version: Access 2000 file format.

You can find the sample here: http://www.rogersaccesslibrary.com/forum/topic558_post573.html

.

Begin Date and End Date from Effective Date

So far in this series on Domain Functions, I've discussed the general syntax (Domain Functions Demystified) and problems involved in building criteria expressions (Domain Functions Demystified: Criteria Expressions). Unfortunately, many of the examples I've given are relatively trivial. So for my next few blog posts, I thought I'd give what I consider truly useful applications of domain functions.

Other Examples:

1. Simulate AutoNumber with DMax
2. Running Sum with DSum
3. Numbered Query with DCount
4. Rolling Average with DAvg and DCount
5. "Difference Between" with DLookup/DMax

Earlier, I looked at the Difference Between query. A similar problem is the BeginDate/EndDate query. Unlike a difference between, which subtracts the value of the previous record from the value of the current record to show the difference, the BeginDate/EndDate query infers the value of the EndDate from the value of the Effective Date in the next record.

For instance, suppose I have a PriceList table where the price for each product is in effect for only a certain date range. But since maintaining a Begin Date and End Date is prone to error, I'd like to simply store an Effective Date in the record. Conceptually, I don't really need both a Begin Date and End Date. A record is in effect from its own Effective Date to the Effective Date (minus 1) of the next row. In other words, consider figure 1below. In row 1, Product 1 is $3 from 1/1/2009 (EffectiveDate) to 12/1/2009 (Effective Date of row 2 minus 1).

So from this data:

Figure 1

I'd like to a create query which would produce following result:

Figure 2

The problem is that SQL does not have positional notation like Excel does. There's no way to simply point to the record following the one you're on. The only way to do it is to somehow identify the next record in terms of the data stored in the record.

For this method to work, I must have a unique record ID. The Autonumber field is ideal for this. It doesn't matter if there are gaps in the sequence, but I have to sort on this field, so there cannot be duplicates and they must be in the order I need displayed. In the above sample, PriceID fits the bill.

The technique is similar to creating a Difference Between Query, but instead of finding the difference between a field on this row and the same field on a previous row, I want to show the value of the field on the NEXT row in the current row.

Over all, I need three steps:

1. Identify the Primary Key of next row.
2. Feed that value to a function (or query) that identifies the next date value,
3. Display the next date value on the current record (manipulating as necessary)

DMax Method

Domain Aggregate functions are an Access-only method to return statistical information about a specific set of records, whether from a table or query. They have three arguments: 1) an expression that identifies a field, 2) a string expression that identifies a domain (that is, the table or query), and 3) a Criteria, which is essentially an SQL Where clause without the word WHERE.

Here are the steps:

I need a DMin function to return the PriceID of the next record in the table:

DMin("PriceID","PriceList","ProductID = " & [ProductID] & " And PriceID > " & [PriceID]

Next, I'll feed that to a DLookup function, which will return the date value from the next row given number of records.

DLookUp("EffectiveDate","PriceList","PriceID = " & DMin("PriceID","PriceList","ProductID = " & [ProductID] & " And PriceID > " & [PriceID]))

Lastly, I'll subtract 1 from [EffectiveDate] of the current record and give the column an alias:

DateAdd("d",-1,DLookUp("EffectiveDate","PriceList","PriceID = " & Nz(DMin("PriceID","PriceList","ProductID = " & [ProductID] & " And PriceID > " & [PriceID]),0))) AS EndDate

The full query, looks like this:

SELECT PriceID, ProductID, EffectiveDate AS BeginDate, DateAdd("d",-1,DLookUp("EffectiveDate","PriceList","PriceID = " & Nz(DMin("PriceID","PriceList","ProductID = " & [ProductID] & " And PriceID > " & [PriceID]),0))) AS EndDate, Price
FROM PriceList
ORDER BY PriceID;

The NZ() function is needed to prevent the last row of a group from displaying an ERROR.

The Order By clause in the query is important. This will sort the query on the PriceID field. I'll need to have that order to use the criteria argument in the DMin.

It is not necessary that the Order By field is an unbroken sequence. As long as that field has unique values and is sorted, it will work.

Figure 3a: Shows the calculated EndDate with NULL for the end date of the last record in the group.

This, of course, accurately represents the data because I don't know the end date of the currently effective price. Having a NULL in the field is most correct from a design standpoint. However, null values are difficult to query, so it is easier from a practical standpoint to put an actual value in the end date that is far in the future. I usually use 12/31/9999 for a future date that's not likely to become obsolete any time soon.

I can easily accomplish this with another NZ function:

SELECT PriceID, ProductID, EffectiveDate AS BeginDate, Nz(DateAdd("d",-1,DLookUp("EffectiveDate","PriceList","PriceID = " & Nz(DMin("PriceID","PriceList","ProductID = " & [ProductID] & " And PriceID > " & [PriceID]),0))),#12/31/9999#) AS EndDate, Price

FROM PriceList

ORDER BY PriceID;

Figure 3b: Shows the calculated EndDate with an artificial end date far in the future in the last record of the group.

Subquery and Outer Join Methods

This query can also be done with a correlated subquery or with an Outer Join, which I may discuss at a later date. However, you can find all three methods on my website in this sample: BeginDateEndDateQuery.mdb.

.

New Sample: Mode.mdb

by Roger Carlson

Access has no Mode function, so you have to write one of your own. This sample database shows how to do that.

You use the function much like the built-in Domain functions (DLookUp, DMax, and so on). That is, you must provide the 1) field name, 2) table name, and 3) a 'Where' Criteria. When used in an aggregate query (see below) you MUST add each field 'grouped by' into the Where Criteria.

See this for more on Domain Aggregate functions see Domain Functions Demystified.

You can find the sample here: http://www.rogersaccesslibrary.com/forum/modemdb-intermediate_topic557.html

NewSample: Query_GrpSequentialsAndMissing

by AD Tejpal

This sample db demonstrates identification of data blocks as well as missing portions in a sequential series. Three types of source data are covered:


(a) Number series.

(b) Alpha-numeric series.

(c) Date series.

Subform on left shows all records, duly highlighting the start of each new block of sequential series. Subform at right depicts group-wise gist (for each PersonID) of sequential blocks as well as missing portions, duly indicating the start and end values for each set. Missing blocks are highlighted in light grey.

Version: Access 2000 file format.
 
You can find the sample here: http://www.rogersaccesslibrary.com/forum/query-grpsequentialsandmissing_topic556.html
 
.

New Sample: TableNormalizationByPureSQL

by A.D. Tejpal

This sample db demonstrates pure query based solution for normalization of data held by a non-normalized table (T_Source) and posting the converted contents to destination table (T_Normalized).


Steps:

1 - Create the empty destination table T_Normalized with a structure identical to that of table T_Source, but without the non-normalized fields (e.g. Red, Green, Blue, Yellow in this sample).

2 - Add two new fields to the newly created table T_Normalized. One meant for holding the names of non-normalized fields and the other for holding corresponding values. In the current sample, these two fields are named Color and Stock respectively.

3 - Create an auxiliary table named T_SourceFieldsConverted having a single field meant for holding names of non-normalized fields. In the current sample, this field is named Color. Populate this table with the names of non-normalized fields (i.e. Red, Green, Blue, Yellow in this sample).

4 - Execution of append query Q_AppNormalized will populate destination table T_Normalized with normalized data, duly converted from source table T_Source. This query is based upon Cartesian join between tables T_SourceFieldsConverted and T_Source.

Version: Access 2000 file format.
 
You can find the sample here: http://www.rogersaccesslibrary.com/forum/tablenormalizationbypuresql_topic554.html
 
.

Domain Function Example: Rolling Average in Query

So far in this series on Domain Functions, I've discussed the general syntax (Domain Functions Demystified) and problems involved in building criteria expressions (Domain Functions Demystified: Criteria Expressions). Unfortunately, many of the examples I've given are relatively trivial. So for my next few blog posts, I thought I'd give what I consider truly useful applications of domain functions.

Other Examples:

1. Simulate AutoNumber with DMax
   
2. Running Sum with DSum
3. Numbered Query with DCount
4. "Difference Between" with DLookup/DMax
5. Begin Date and End Date from Effective Date

Another value that is difficult to produce in a query, is the Rolling Average for a given number of records.

For instance, suppose I wanted to display a rolling average for the last 12 weeks for the table below:

Figure1

For Week 26, I need to display the average for weeks 15-26 (39.85). For Week 25, it would be the average for weeks 14-25 (43.85), and so forth. For weeks with less than 12 in the recordset, it will average only those weeks available. So Week 9 would only average weeks 7-9 (53.67).

In other words, this:

Figure 2

The problem is that SQL does not have positional notation like Excel does. There's no way to simply point to the record above the one you're on -- or the previous 12, for that matter. The only way to do it is to somehow identify the previous records in terms of a Where condition. Since this Where condition must be evaluated for each line, O can do this with a domain aggregate function or a correlated subquery. In this case, two domain functions and two subqueries.

For either method to work, I must have a unique record ID. The Autonumber field is ideal for this. It doesn't matter if there are gaps in the sequence, but I have to sort on this field, so there cannot be duplicates and they must be in the order I need displayed. In the above sample, ID fits the bill.

Domain Function Method (DCount and DAvg)

Domain Aggregate functions are an Access-only method to return statistical information about a specific set of records, whether from a table or query. DCount in particular will return the number of records in a given recordset. DAvg will return the average of a given recordset. Both functions have three arguments: 1) an expression that identifies a field, 2) a string expression that identifies a domain (that is, the table or query), and 3) a Criteria, which is essentially an SQL Where clause without the word WHERE.

The first step in this process is to create an unbroken sequence number for the records. It must be unbroken so I can subtract 12 from it to average the correct number of weeks. The second step produces the average.

Step1: DCount_RollingAverage1:

SELECT DCount("ID","Table1","ID <=" & [ID]) AS Sequence, tWeek, tValue
FROM Table1
ORDER BY ID DESC;

The Order By clause in the query is important. This will sort the query on the ID field. I'll need to have that order to use the criteria argument in the DCount.

Here's how it works.

For each record in the query, Access runs the DCount function. The DCount returns the number of records in the domain where the ID in the function is less than or equal to the ID in that record of the query.

So in the first record, the ID is 1. So the DCount opens the domain (essentially opens the Customers table again) and it sees that there is only 1 record whose ID is less than or equal to 1. So it returns 1.

Then it processes the second record. The ID of that record is 3, and the DCount function sees that there are only 2 records which have an ID whose value is less than or equal to 2. So it returns 2.

It is not necessary that the Order By field is an unbroken sequence. As long as that field has unique values and is sorted, it will work.

 Figure 3

Step2: DCount_RollingAverage2:

Now that DCount_RollingAverage1is a recordset with an unbroken sequence, I can use as it as the record source for the query that will create the rolling averages:

SELECT Sequence, tWeek, tValue, DAvg("tValue","[DCount_RollingAverage1]",
"Sequence Between " & [Sequence] & " And " & [Sequence]-12) AS [12-Week Rolling Average]
FROM DCount_RollingAverage1;

For each record in the query, Access runs the DAvg function. The DAvg returns the average for the range of values between the sequence number and the sequence number minus 12.

So in the first record, the Sequence is 26. So the DAvg opens the domain (essentially opens Table1 again) and averages weeks 15-26. Then it processes the second record, averaging weeks 14-25 and so forth.

Figure 4

Subquery Method

There is no way to combine these two queries into one. To do that, I'd need to use a correlated subquery, which I may discuss at a later date. However, you can find both methods on my website in this sample: RollingAverages.mdb.

Domain Function Example: "Difference Between" in Query

So far in this series on Domain Functions, I've discussed the general syntax (Domain Functions Demystified) and problems involved in building criteria expressions (Domain Functions Demystified: Criteria Expressions). Unfortunately, many of the examples I've given are relatively trivial. So for my next few blog posts, I thought I'd give what I consider truly useful applications of domain functions.

Other Examples:

1. Simulate AutoNumber with DMax
2. Running Sum with DSum
3. Numbered Query with DCount
4. Rolling Average with DAvg and DCount
5. Begin Date and End Date from Effective Date

Earlier, I looked at the Running Sum query. A similar (but opposite) problem is the Difference Between query. Unlike a running sum, which adds the value of a field in a record to the value of the same field in the previous record, the "difference between" query subtracts the value of the previous record from the value of the current record to show the difference.

For instance, suppose I wanted to display the difference in days between orders in the table below:

 

Figure1: Need to calculate the difference between records.

The difference in days between records 1 and 2 is 4, between 2 and 3 is 7, between 3 and 4 is -22, and so forth.

The problem is that SQL does not have positional notation like Excel does. There's no way to simply point to the record above the one you're on. The only way to do it is to somehow identify the previous record in terms of a Where condition. Since this Where condition must be evaluated for each line, I can do this with a domain aggregate functions (DMax & DLookup).

For this method to work, I must have a unique record ID. The Autonumber field is ideal for this. It doesn't matter if there are gaps in the sequence, but I have to sort on this field, so there cannot be duplicates and they must be in the order I need displayed. In the above sample, OrderDetailsID fits the bill.

The technique is similar to creating a Numbered Query or a Running Sum in a query, but instead of just counting or summing all the records above the current record, I have to find just the previous record. This adds an additional complication, which requires an additional domain function.

Specifically, I need three steps:

1. Use a domain function (DMax) to find the unique identifier the previous row.
2. Then feed that value in to another domain function (DLookup) that identifies the previous value,
3. And then subtract the previous value from the current value.

One way to show the difference between records is over the whole recordset. Later, I'll look at showing the difference between over groups of records.

Difference Between - Over All
Domain Aggregate functions are an Access-only method to return statistical information about a specific set of records, whether from a table or query. They have three arguments: 1) an expression that identifies a field, 2) a string expression that identifies a domain (that is, the table or query), and 3) a Criteria, which is essentially an SQL Where clause without the word WHERE.

Here are the steps:

1. I need a DMax function to return the OrderDetailID of the previous record in the table:
   DMax("OrderDetailID","tblOrderDetails","OrderDetailID < " & [OrderDetailID])
2. Next, I'll feed that to a DLookup function, which will return the date value from the previous row given number of records.
   DLookUp("OrderDate","tblOrderDetails","OrderDetailID = " & DMax("OrderDetailID","tblOrderDetails","OrderDetailID < " & [OrderDetailID]))
3. Lastly, I'll subtract the this value from [OrderDate]of the current record and give the column an alias:[OrderDate]-DLookUp("OrderDate","tblOrderDetails","OrderDetailID = " & DMax("OrderDetailID","tblOrderDetails","OrderDetailID < " & [OrderDetailID])) AS DaysBetween

The full query, looks like this:

SELECT tblOrderDetails.OrderDetailID, tblOrderDetails.OrderID, tblOrderDetails.OrderDate, nz([OrderDate]-DLookUp("OrderDate","tblOrderDetails","OrderDetailID = " & nz(DMax("OrderDetailID","tblOrderDetails","OrderDetailID < " & [OrderDetailID]),0)),0) AS DaysBetween

FROM tblOrderDetails

ORDER BY tblOrderDetails.OrderDetailID;

The two NZ() functions are needed to display a zero on the first line. Otherwise, it would return an ERROR.

The Order By clause in the query is important. This will sort the query on the OrderDetailID field. I'll need to have that order to use the criteria argument in the DMax.

It is not necessary that the Order By field is an unbroken sequence. As long as that field has unique values and is sorted, it will work.

Figure 2: Shows the time difference in days between subsequent records.

Difference Between - Over Group
 SELECT tblOrderDetails.OrderDetailID, tblOrderDetails.OrderID, tblOrderDetails.OrderDate,

Nz([OrderDate]-DLookUp("OrderDate","tblOrderDetails","OrderDetailID = " & Nz(DMax("OrderDetailID","tblOrderDetails","OrderID = " & [OrderID] & " And OrderDetailID < " & [OrderDetailID]),0)),0) AS DaysBetween

FROM tblOrderDetails

ORDER BY tblOrderDetails.OrderDetailID;

Figure 3: As the OrderID changes, the DaysBetween resets to zero.

The only difference between this query and the OverAll query is the addition of

"OrderID = " & [OrderID]

to the "Where" condition of the DMax function. This sets the value of the first record of each group to zero.

Subquery Method

As I said, these can also be done with a correlated subquery, which I may discuss at a later date. However, you can find both methods on my website in this sample: DaysBetween.mdb