Added a new failure type 'ORDERBY_FAILURE' #260
Conversation
|
Chun, Can you review these changes? These changes introduce a new failure type for statements with order-by clauses. Current tests that fail have been grandfathered. New tests that fail will be flagged. |
…ement has an order-by clause that cannot be validated by the Test Framework. Please see the README.md file for information on how to construct a SQL statement with an order-by clause that can be validated by the Test Framework. Added a new verification method, "text", which checks if the actual output and expected output are exactly the same, byte by byte. It verifies content as well as order. Every row that is returned must be returned in the same order as in the expected output. Added a whitelist file for orderby tests. Tests that are listed in this file are SQL statements with an order-by clause. The order-by clause cannot be validated. Instead of marking these tests as failed, they are PASSED until they can be fixed in the future. There are 302 tests that are currently failing at the time of this commit. They will need to be fixed at some piont in the future.
|
I'm not sure if that's the right approach, Robert. If we don't do this now - it's likely that this will never happen - we have many such instances which are still pending today. It's better we wait to get the fix in until all tests are modified. -1 for now. Let's discuss more. |
|
If we don't exposed orderby failures now, then when we add new orderby tests, we cannot determine if the orderby is working or not. I'm not sure if this is a good idea. Are there other cases where we think tests are passing, and they are actually failing? Sure, let's discuss more. |
|
No, I do think we should be fixing this. All I'm saying is that lets not defer updating dependent tests. In my opinion, lets fix the tests you identified first (if a change to baseline is required) and then get your fix in. |
|
Okay, sorry, I mis-understood. Ideally, I would like to see all the tests fixed. I don't know how long that will take. This isn't a priority, and I still have 1.10 work to do, so I'm not spending much time on this at the moment. This is my last planned fix for the time being, until I get my 1.10 work done. I am just trying to get this issue to the point where it is stable, meaning it doesn't get any worse. Anyway, I'm fine with not putting this in for now. At least we are aware of the issue. |
|
Sure. Let's discuss this in person next week when I'm back. |
Agirish
changed the title
|
@cchang738, the discussion is still pending. Both Robert and I are busy with other tasks, so we'll look into this in a few weeks. Hopefully sooner. |
Which occurs if a SQL statement has an order-by clause that cannot be validated by the Test Framework. Please see the README.md file for information on how to construct a SQL statement with an order-by clause that can be validated by the Test Framework.
Added a new verification method, "text", which checks if the actual output and expected output are exactly the same, byte by byte. It verifies content as well as order. Every row that is returned must be returned in the same order as in the expected output.
Added a whitelist file for orderby tests. Tests that are listed in this file are SQL statements with an order-by clause. The order-by clause cannot be validated. Instead of marking these tests as failed, they are PASSED until they can be fixed in the future. There are 302 tests that are currently
failing at the time of this commit. They will need to be fixed at some point in the future.