Order of precedence in where clause
Im getting an error on this statement:
SELECT *
FROM
[My Table]
WHERE
ISDATE([DATE OF
BIRTH])
= 1
AND
CONVERT(DATETIME,
[DATE OF BIRTH], 101)
<
'01/01/1899'
Because its evaluating the second condition first and running into non dates. Is there a way to change this where clause to guarantee it will exclude the non-dates? Also, because of the way the app was written with dynamic SQL, it has to be done
in the where clause.
the error is :
The conversion of a nvarchar data type to a datetime data type resulted in an out-of-range value.
December 19th, 2013 8:15pm
Yes.
Make the ISDATE condition a subquery or CTE.
December 19th, 2013 8:18pm
The optimiser can change the order of execution.
try
where case when isdate([date of birth]) = 1 then
CONVERT(DATETIME,
[DATE OF BIRTH], 101)
else '18990101' end < '18990101'
That's if you can rely on isdate to guarantee a valid date conversion.
December 19th, 2013 8:19pm
The optimiser can change the order of execution.
try
where case when isdate([date of birth]) = 1 then
CONVERT(DATETIME,
[DATE OF BIRTH], 101)
else '18990101' end < '18990101'
That's if you can rely on isdate to guarantee a valid date conversion.
- Marked as answer by
jschroeder
Thursday, December 26, 2013 3:04 PM
December 19th, 2013 8:19pm
Thanks
December 19th, 2013 8:23pm
I also don't think that subquery will actually help. The optimizer can still re-arrange execution. Only if you subselect into a temp table first.
December 19th, 2013 8:50pm
December 19th, 2013 8:54pm
I'm sorry, but a subquery or CTE is not the answer here. The optimizer can, at will, do "predicate pushing" (Bingle it for more info) and push the predicate of the outer query into the inner query and you are back to the exact same execution plan as you
began with.
Use Naomi's suggestion to materialize the "safe" rows into a table variable or temp table first and then query that from a second query, or go with Nigel's suggestion (a CASE construct). Your tests will tell you which performs best in your particular situation.
December 19th, 2013 9:40pm
BTW, I tried unmarking Kalman's reply as an answer and it didn't work for me. I have a bug reported in forum's issues because lately I am unable to unpropose or unmark answers.
December 19th, 2013 9:59pm
Have you considered writing SQL and not dialect? The first thing you missed is that we do not embed blanks in data element names; read ISO-11179 or any book on data modeling. The only date display format allowed in ANSI/ISO Standard SQL is ISO-8601, which is
yyyy-mmn-dd; it is one of the most common ISO standards in IT, as it is embedded in so many other standards.
We do not use the 1970's Sybase string function CONVERT; we have a proper DATE data type now, so dates are not COBOL strings any more! The ANSI/ISO Standard CAST (birth_date AS DATE) will do the conversion, if your data is screwed up. The DATE data type has
a range of '0001-01-01' to '9999-12-31', which should be enough for anyone but a museum or an archeologist. Oh, it is also smaller than the old proprietary Sybase DATETIME data type.
We do not use things like ISDATE() in a correctly build system; we know the data type of the column and have done the data scrubbing with ETL tools or a presentation layer. Do you understand the concept of a tiered architecture?
The data in the tables is supposed to be clean.
What you are getting are kludges and not real help. Your approach is to write 1970's COBOL style code in 1970's Sybase SQL. This will cost you performance, maintainability, portability and the cool kids will laugh at you.
December 19th, 2013 10:26pm
The only date display format allowed in ANSI/ISO Standard SQL is ISO-8601, which is yyyy-mmn-dd; it is one of the most common ISO standards in IT, as it is embedded in so many other standards.
'yyyy-mm-dd' is NOT the correct choice for SQL server though
Have a read of this extremely helpful article: http://www.karaszi.com/sqlserver/info_datetime.asp (in particular the
Warnings and common misconceptions
section)
- Proposed as answer by
Naomi NModerator
Thursday, December 19, 2013 7:55 PM
December 19th, 2013 10:53pm
The only date display format allowed in ANSI/ISO Standard SQL is ISO-8601, which is yyyy-mmn-dd; it is one of the most common ISO standards in IT, as it is embedded in so many other standards.
'yyyy-mm-dd' is NOT the correct choice for SQL server though
Have a read of this extremely helpful article: http://www.karaszi.com/sqlserver/info_datetime.asp (in particular the
Warnings and common misconceptions
section)
- Proposed as answer by
Naomi NModerator
Thursday, December 19, 2013 7:55 PM
- Unproposed as answer by
Pan ZhangMicrosoft contingent staff, Administrator
Tuesday, December 24, 2013 8:25 AM
- Proposed as answer by
Pan ZhangMicrosoft contingent staff, Administrator
Tuesday, December 24, 2013 8:26 AM
December 19th, 2013 10:53pm
Exactly. I was meaning to point this out as well.
December 19th, 2013 10:56pm
Celko,
|dates are not COBOL strings any more
As a matter of fact, yes the date is a COBAL string. It's being imported from a COBAL system.
This link may help you http://en.wikipedia.org/wiki/Pedant
Also, since I inherited and did not write the app, my options are limited.
|The first thing you missed is that we do not embed blanks in data element names
The field name is from a COBAL system written decades ago.
| the cool kids will laugh at you.
I am a cool kid, unlike you. And I'm the one laughing.
December 20th, 2013 1:21am
I cannot rely on isdate to guarantee a valid date conversion.
December 20th, 2013 2:27am
ISDATE will tell you if the value can be converted into a date using one of the conversion styles. If you have mixed styles saved in the database, then you're in trouble.
December 20th, 2013 2:40am
Not using standards is never right. Microsoft is moving to ANSI/ISO Standard temporal data. This means that Nobody should be using the old 1970's Sybase DATETIME data types. The CAST() and future temporal features are all going to be based on the standards,
with the old stuff hanging on as a family curse that will have to be kludged constantly. Not portable, not maintainable, not readable to new programmers that did not grow up with 1970's Sybase SQL. Hardly what I would call a best choice.
The bad news is that we are stuck with DATETIME2 instead of the correct TIMESTAMP. Just like Oracle and its VARCHAR2() from their initial screwup.
All of these statements return '2003-02-28', as per ANSI/ISO.
SET LANGUAGE us_english
SELECT CAST('2003-02-28' AS DATETIME2);
SET LANGUAGE british
SELECT CAST('2003-02-28' AS DATETIME2);
SET LANGUAGE us_english
SELECT CAST('02/28/2003' AS DATETIME2);
SET LANGUAGE british
SELECT CAST('28/02/2003' AS DATETIME2);
December 20th, 2013 5:41am
Let me state the obvious, that the best thing to do, is to change the data type of "DATE OF BIRTH" to date. When unknown, store NULL.
Then, when filtering on "DATE OF BIRTH", compare against a safe date format, which would be 'YYYYMMDD'.
This and that would lead to:
SELECT my_column
FROM "My Table"
WHERE "DATE OF BIRTH" < '18990101'
If you need the query to actually clean up your table, then just take it step by step, and use temporary tables to process the rows and clean the values.
December 20th, 2013 10:37am
Not using standards is never right.
Are you sugesting I rebuild the entire application I've inhereted? I don't think that's in the company budget for this year. "Have you considered writing" useful replies instead of making normative judgments? Oh, and using this forum as a
vehicle for self promotion is "never right" e
December 20th, 2013 10:47am
Are you suggesting I rebuild the entire application I've inherited? is classic! Lets close our eyes and just kludge it (with help from SQL forums) until it falls apart with deprecation.
You will have switched job by then anyway, right?
This sounds like you inherited a classic family curse application written in SQL Server 2000 or earlier and never given adaptive maintenance as releases came out. Say that in your positing! I
am suggesting that you begin a policy of SEI style professionalism. Do they still teach software maintenance as part of Software Engineering?
Corrective: maintenance performed to correct faults in hardware or software. Oh, you meant (x >= 0) and and not (x > 0) and so forth. This one explains itself and it is what most people think of when they hear the words
software maintenance; it is not the whole picture.
Adaptive: software maintenance performed to make a computer program usable in a changed
environment. Us database guys have it a little easier because in a tiered environment, much of this is done in the presentation layers. Our job is to throw data over the wall in industry standard format, so that any display
formatting, special computations, etc can be done by someone else. But if the environment changes, as when European countries switched to the Euro or US retail went to GTIN from UPC, then we have to change our tables.
We also have to re-write code when features are deprecated. Anyone want ot try to use *= in place of LEFT OUTER JOIN?
Perfective: software maintenance performed to improve the performance, maintainability, or other
attributes of a computer program. Functionality is the same. In a procedural language, we might replace a Bubble Sort with a QuickSort.. Better queries and better indexes are part of how we do it in SQL.
Preventative: maintenance performed for the purpose of preventing problems before they occur. This is the red-haired stepchild of software maintenance. Have you worked with people who like disaster? They love to run in and save
the day, be a hero. But they could have done something when it was easy and never had the problem.
In SQL, we have constraints that can prevent bad data. How many of your INTEGER columns in your schema have CHECK (x >= 0) or CHECK (x > 0) constraints to help the optimizer? This should cost you a simple ALTER TABLE.
If you know how, you can set up changes with a maintenance program and get the system corrected.
Also why do you think that professionalism and standards are "self-promotion"? I promise that I get no kick-back from ISO when you read a standard :)
December 20th, 2013 1:47pm
Have you considered writing SQL and not dialect?
Have you?
December 20th, 2013 2:48pm
LOL! very instructive. John, I've had this issue multiple time the solution proposed by Nigel is the one which worked for me all the time.
Statements inside a case are executed sequentially.
"The CASE statement evaluates its conditions sequentially and stops with the first condition whose condition is satisfied."
You must however make sure you do not put potentially erroneous statements in the when:
"In some situations, an expression is evaluated before a CASE statement receives the results of the expression as its input. Errors in evaluating these expressions are possible. Aggregate expressions that appear in WHEN arguments to a CASE statement
are evaluated first, then provided to the CASE statement. For example, the following query produces a divide by zero error when producing the value of the MAX aggregate. This occurs prior to evaluating the CASE expression."
Take a look at the remark section of this article:
http://technet.microsoft.com/en-us/library/ms181765.aspx
In other words, if your date manipulations are inside the then statements and you tested the format in the when statements then you should be OK.
- Edited by
Antoine F
12 hours 55 minutes ago
December 20th, 2013 5:32pm
LOL! very instructive. John, I've had this issue multiple time the solution proposed by Nigel is the one which worked for me all the time.
Statements inside a case are executed sequentially.
"The CASE statement evaluates its conditions sequentially and stops with the first condition whose condition is satisfied."
You must however make sure you do not put potentially erroneous statements in the when:
"In some situations, an expression is evaluated before a CASE statement receives the results of the expression as its input. Errors in evaluating these expressions are possible. Aggregate expressions that appear in WHEN arguments to a CASE statement
are evaluated first, then provided to the CASE statement. For example, the following query produces a divide by zero error when producing the value of the MAX aggregate. This occurs prior to evaluating the CASE expression."
Take a look at the remark section of this article:
http://technet.microsoft.com/en-us/library/ms181765.aspx
In other words, if your date manipulations are inside the then statements and you tested the format in the when statements then you should be OK.
- Edited by
Antoine F
Friday, December 20, 2013 10:30 PM
December 21st, 2013 1:23am
ANSI/ISO Standard SQL is all I write until I am cornered by a SQL which is not up to speed yet. And even then I put the correct code in a comment, so that the poor bastards who come after me can uncomment and test with the next
release.
December 23rd, 2013 1:29am
>> As a matter of fact, yes the date is a COBAL [sic] string. It's being imported from a COBAL [sic] system. <<
You misspelled the name of the language :)
>> Also, since I inherited and did not write the app, my options are limited. <<
The options are much better than they used to be. There are ETL tools, etc. But if all you have is COBOL, it is really good at string manipulations.
>> { we do not embed blanks in data element names } The field name is from a COBAL [sic] system written decades ago.<<
COBOL uses dashes where SQL uses underscores. That could be a bulk text edit if you need to keep pre-ISO-11179 names. But there is no excuse for not having a DATE column to which to map the COBOL field.
December 23rd, 2013 1:37am
CASE expression, not statement!
This was trickier than people first think. When we were voting on this, we found that we had to evaluate
all of the THEN and ELSE clauses to determine the data type of the expression. But lead to problems with levels of aggregation and forces the THEN and ELSE clauses to be at the same level. ARRGH!
December 23rd, 2013 1:44am
The only date display format allowed in ANSI/ISO Standard SQL is ISO-8601, which is yyyy-mmn-dd; it is one of the most common ISO standards in IT
SET LANGUAGE british
SELECT CAST('2013-05-21' AS DATETIME)
returns:
Msg 242, Level 16, State 3, Line 2
The conversion of a varchar data type to a datetime data type resulted in an out-of-range value.
Contrary to popular belief, YYYY-MM-DD
for DATETIME
variables is
NOT language-/dateformat-independent in SQL Server! :)
yyyymmdd is the best format in SQL Server.
Thank you for correcting my spelling.
December 24th, 2013 2:03am
The optimiser can change the order of execution.
try
where case when isdate([date of birth]) = 1 then
CONVERT(DATETIME,
[DATE OF BIRTH], 101)
else '18990101' end < '18990101'
That's if you can rely on isdate to guarantee a valid date conversion.
Thanks Nigel. A variation on this works for me.
December 26th, 2013 6:04pm
'yyyy-mm-dd' is NOT the correct choice for SQL server though
Have a read of this extremely helpful article: http://www.karaszi.com/sqlserver/info_datetime.asp (in particular the
Warnings and common misconceptions
section)
Instead of referring to an article it is better to state your point.
Why is 'YYYY-MM-DD' ANSI date string not good for SQL Server?
I am looking for a proof not hearsay. After all we are doing Computer Science, aren't we?
Tibor stated the same in another thread (without explanation), but I don't find an explanation in the referenced article.
'YYYYMMDD' & 'YYYY-MM-DD' are both valid ANSI date strings.
Hi Tibor,
>I'm sorry, but a subquery or CTE is not the answer here.
Can you provide a proof? Thanks.
December 26th, 2013 6:18pm
Did you check the article I suggested (the article by Saeid Hasani)?
As for Tibor's blog, it provides a sample of yyyy-mm-dd format failing and I just demonstrated it to you in another thread.
December 26th, 2013 8:05pm
Hi Naomi,
Tibor is not Microsoft. The 'YYYY-MM-DD' format is all over BOL and SSMS.
I would not stop using it because it converts differently in some foreign languages when proper conversion style number is not specified.
It converts OK in USA.
Many things are different in date/datetime string representations when dealing with foreign languages, most famous is dmy UK usage as opposed to mdy USA usage.
December 26th, 2013 8:19pm
The point is that if we know that particular format may not convert correctly when converting to datetime, then why keep suggesting that wrong not 100% safe format when there is one 100% safe?
In my code samples I never use yyyy-mm-dd format answering questions and in our code which I was code reviewing recently I also suggested that safe format to my colleague, so he went ahead and fixed it.
December 26th, 2013 8:35pm
This is a format used for presentation, not the internal format and not the format that needs to be used for datetime constants that do not have time portion.
Did you find in BOL the implicit constant of yyyy-mm-dd representing datetime value? And even if you do find it, it doesn't mean BOL shows it correctly.
December 26th, 2013 9:32pm
This was a question about order precedence. Why are you blabbering about about what's taught in software engineering classes? Frankly, myself and a lot of other people wish you would shut the F up.
February 12th, 2014 6:50pm
This was a question about order precedence. Why are you blabbering about about what's taught in software engineering classes? Frankly, myself and a lot of other people wish you would shut
February 13th, 2014 1:33am