Bug/date not recognized (#54)

* MagicBot/add-model-counts updates

* bug/date-not-recognized

* revisions

* update ymls & add tests

* regen docs

* update for next vs last year

* regen docs

* delete .gitkeep

* Update readme and changelog

CHANGELOG.md

@@ -1,3 +1,21 @@
+# dbt_xero v0.7.0
+[PR #54](https://github.com/fivetran/dbt_xero/pull/54) includes the following updates:
+
+## Breaking Changes
+- Corrected the calculation of `current_year_end_date` in the `xero__balance_sheet_report` model. Previously, `current_year_end_date` was miscalculated in certain scenarios, impacting the classification of records with the `account_name` value "Retained Earnings."
+  - This is labeled as a breaking change since it may affect prior labels assigned. We recommend reviewing your records to ensure they align with this corrected logic.
+
+## Bug Fixes
+- Updated the `xero__balance_sheet` model to resolve a run error when an organization's financial year end date is February 29.
+
+## Documentation
+- Updated README formatting for consistency with other packages.
+- Added Quickstart model counts to README. ([#52](https://github.com/fivetran/dbt_xero/pull/52))
+- Corrected references to connectors and connections in the README. ([#52](https://github.com/fivetran/dbt_xero/pull/52))
+
+## Under the Hood (maintainers only)
+- Added consistency tests for the end models.
+
 # dbt_xero v0.6.2
 [PR #46](https://github.com/fivetran/dbt_xero/pull/46) includes the following updates:
 

README.md

@@ -1,4 +1,6 @@
-<p align="center">
+# Xero Transformation dbt Package ([Docs](https://fivetran.github.io/dbt_xero/))
+
+<p align="left">
     <a alt="License"
         href="https://github.com/fivetran/dbt_xero/blob/main/LICENSE">
         <img src="https://img.shields.io/badge/License-Apache%202.0-blue.svg" /></a>
@@ -8,9 +10,11 @@
         <img src="https://img.shields.io/badge/Maintained%3F-yes-green.svg" /></a>
     <a alt="PRs">
         <img src="https://img.shields.io/badge/Contributions-welcome-blueviolet" /></a>
+    <a alt="Fivetran Quickstart Compatible"
+        href="https://fivetran.com/docs/transformations/dbt/quickstart">
+        <img src="https://img.shields.io/badge/Fivetran_Quickstart_Compatible%3F-yes-green.svg" /></a>
 </p>
 
-# Xero Transformation dbt Package ([Docs](https://fivetran.github.io/dbt_xero/))
 ## What does this dbt package do?
 - Produces modeled tables that leverage Xero data from [Fivetran's connector](https://fivetran.com/docs/applications/xero) in the format described by [this ERD](https://fivetran.com/docs/applications/xero#schemainformation) and builds off the output of our [Xero source package](https://github.com/fivetran/dbt_xero_source).
 
@@ -28,14 +32,16 @@ The following table provides a detailed list of all tables materialized within t
 | [xero__balance_sheet_report](https://github.com/fivetran/dbt_xero/blob/main/models/xero__balance_sheet_report.sql)    | Each record represents the state of the balance sheet for a given account on a given month.                            |
 | [xero__invoice_line_items](https://github.com/fivetran/dbt_xero/blob/main/models/xero__invoice_line_items.sql)      | Each record represents an invoice line item enriched with the account, contact, and invoice information.                   |
 
+### Materialized Models
+Each Quickstart transformation job run materializes 22 models if all components of this data model are enabled. This count includes all staging, intermediate, and final models materialized as `view`, `table`, or `incremental`.
 <!--section-end-->
 
 ## How do I use the dbt package?
 
 ### Step 1: Prerequisites
 To use this dbt package, you must have the following:
 
-- At least one Fivetran Xero connector syncing data into your destination.
+- At least one Fivetran Xero connection syncing data into your destination.
 - A **BigQuery**, **Snowflake**, **Redshift**, **PostgreSQL**, or **Databricks** destination.
 
 ### Step 2: Install the package
@@ -44,7 +50,7 @@ Include the following xero package version in your `packages.yml` file:
 ```yaml
 packages:
   - package: fivetran/xero
-    version: [">=0.6.0", "<0.7.0"] # we recommend using ranges to capture non-breaking changes automatically
+    version: [">=0.7.0", "<0.8.0"] # we recommend using ranges to capture non-breaking changes automatically
 ```
 Do NOT include the `xero_source` package in this file. The transformation package itself has a dependency on it and will install the source package as well.
 ### Step 3: Define database and schema variables
@@ -63,8 +69,8 @@ vars:
 
 If you are using multi-currency accounting in Xero, you are likely to have unrealized currency gains as part of your profit and loss statement. These gains/losses do not exist within the actual journals in Xero. As a result, you will find that those lines are missing from the outputs of this package. All realised currency gains will be present and your balance sheet will still balance.
 
-#### Unioning Multiple Xero Connectors
-If you have multiple Xero connectors in Fivetran and would like to use this package on all of them simultaneously, we have provided functionality to do so. The package will union all of the data together and pass the unioned table into the transformations. You will be able to see which source it came from in the `source_relation` column of each model. To use this functionality, you will need to set **either** (**note that you cannot use both**) the `union_schemas` or `union_databases` variables:
+#### Unioning Multiple Xero Connections
+If you have multiple Xero connections in Fivetran and would like to use this package on all of them simultaneously, we have provided functionality to do so. The package will union all of the data together and pass the unioned table into the transformations. You will be able to see which source it came from in the `source_relation` column of each model. To use this functionality, you will need to set **either** (**note that you cannot use both**) the `union_schemas` or `union_databases` variables:
 
 ```yml
 # dbt_project.yml

dbt_project.yml

@@ -1,5 +1,5 @@
 name: 'xero'
-version: '0.6.2'
+version: '0.7.0'
 config-version: 2
 require-dbt-version: [">=1.3.0", "<2.0.0"]
 vars:

integration_tests/dbt_project.yml

@@ -1,7 +1,8 @@
 name: 'xero_integration_tests'
-version: '0.6.2'
+version: '0.7.0'
 profile: 'integration_tests'
 config-version: 2
+
 vars:
   xero_source:
     xero_account_identifier: "xero_account_data"
@@ -15,6 +16,9 @@ vars:
     xero_credit_note_identifier: "xero_credit_note_data"
     xero_schema: xero_integration_tests_2
 
+models:
+  +schema: "xero_{{ var('directed_schema','dev') }}"
+
 dispatch:
   - macro_namespace: dbt_utils
     search_order: ['spark_utils', 'dbt_utils']

integration_tests/tests/consistency/consistency_balance_sheet_report.sql

@@ -0,0 +1,45 @@
+{{ config(
+    tags="fivetran_validations",
+    enabled=var('fivetran_validation_tests_enabled', false)
+) }}
+
+with prod as (
+    select *
+    from {{ target.schema }}_xero_prod.xero__balance_sheet_report
+),
+
+dev as (
+    select *
+    from {{ target.schema }}_xero_dev.xero__balance_sheet_report
+), 
+
+prod_not_in_dev as (
+    -- rows from prod not found in dev
+    select * from prod
+    except distinct
+    select * from dev
+),
+
+dev_not_in_prod as (
+    -- rows from dev not found in prod
+    select * from dev
+    except distinct
+    select * from prod
+),
+
+final as (
+    select
+        *,
+        'from prod' as source
+    from prod_not_in_dev
+
+    union all -- union since we only care if rows are produced
+
+    select
+        *,
+        'from dev' as source
+    from dev_not_in_prod
+)
+
+select *
+from final

integration_tests/tests/consistency/consistency_general_ledger.sql

@@ -0,0 +1,45 @@
+{{ config(
+    tags="fivetran_validations",
+    enabled=var('fivetran_validation_tests_enabled', false)
+) }}
+
+with prod as (
+    select *
+    from {{ target.schema }}_xero_prod.xero__general_ledger
+), 
+
+dev as (
+    select *
+    from {{ target.schema }}_xero_dev.xero__general_ledger
+), 
+
+prod_not_in_dev as (
+    -- rows from prod not found in dev
+    select * from prod
+    except distinct
+    select * from dev
+),
+
+dev_not_in_prod as (
+    -- rows from dev not found in prod
+    select * from dev
+    except distinct
+    select * from prod
+),
+
+final as (
+    select
+        *,
+        'from prod' as source
+    from prod_not_in_dev
+
+    union all -- union since we only care if rows are produced
+
+    select
+        *,
+        'from dev' as source
+    from dev_not_in_prod
+)
+
+select *
+from final

integration_tests/tests/consistency/consistency_invoice_line_items.sql

@@ -0,0 +1,45 @@
+{{ config(
+    tags="fivetran_validations",
+    enabled=var('fivetran_validation_tests_enabled', false)
+) }}
+
+with prod as (
+    select *
+    from {{ target.schema }}_xero_prod.xero__invoice_line_items
+), 
+
+dev as (
+    select *
+    from {{ target.schema }}_xero_dev.xero__invoice_line_items
+), 
+
+prod_not_in_dev as (
+    -- rows from prod not found in dev
+    select * from prod
+    except distinct
+    select * from dev
+),
+
+dev_not_in_prod as (
+    -- rows from dev not found in prod
+    select * from dev
+    except distinct
+    select * from prod
+),
+
+final as (
+    select
+        *,
+        'from prod' as source
+    from prod_not_in_dev
+
+    union all -- union since we only care if rows are produced
+
+    select
+        *,
+        'from dev' as source
+    from dev_not_in_prod
+)
+
+select *
+from final

integration_tests/tests/consistency/consistency_profit_and_loss_report.sql

@@ -0,0 +1,45 @@
+{{ config(
+    tags="fivetran_validations",
+    enabled=var('fivetran_validation_tests_enabled', false)
+) }}
+
+with prod as (
+    select *
+    from {{ target.schema }}_xero_prod.xero__profit_and_loss_report
+), 
+
+dev as (
+    select *
+    from {{ target.schema }}_xero_dev.xero__profit_and_loss_report
+), 
+
+prod_not_in_dev as (
+    -- rows from prod not found in dev
+    select * from prod
+    except distinct
+    select * from dev
+),
+
+dev_not_in_prod as (
+    -- rows from dev not found in prod
+    select * from dev
+    except distinct
+    select * from prod
+),
+
+final as (
+    select
+        *,
+        'from prod' as source
+    from prod_not_in_dev
+
+    union all -- union since we only care if rows are produced
+
+    select
+        *,
+        'from dev' as source
+    from dev_not_in_prod
+)
+
+select *
+from final

models/xero__balance_sheet_report.sql

@@ -10,21 +10,35 @@ with calendar as (
 
 ), organization as (
 
-    select *
+    select 
+        *,
+        cast(extract(year from current_date) as {{ dbt.type_string() }}) as current_year,
+        cast(extract(year from {{ dbt.dateadd('year', 1, 'current_date') }}) as {{ dbt.type_string() }}) as next_year
     from {{ var('organization') }}
 
+
 ), year_end as (
 
+-- Calculate the current financial year-end date for each organization:
+-- For February, determine last day by subtracting 1 day from March 1, avoiding leap year logic.
+-- Compare the year end date to the current date:
+--   Use this year's date if it's on or after the current date.
+--   Otherwise, use the next year's corresponding date.
     select 
-        case
-            when cast(extract(year from current_date) || '-' || financial_year_end_month || '-' || financial_year_end_day as date) >= current_date
-            then cast(extract(year from current_date) || '-' || financial_year_end_month || '-' || financial_year_end_day as date)
-            else case when financial_year_end_month = 2 and financial_year_end_day = 29 
-                then cast(extract(year from {{ dbt.dateadd('year', -1, 'current_date') }}) || '-' || financial_year_end_month || '-28' as date) -- Necessary for organizations with a reported fiscal year end of 02-29 as the previous year will not be a leap year and must be the 28th. 
-                else cast(extract(year from {{ dbt.dateadd('year', -1, 'current_date') }}) || '-' || financial_year_end_month || '-' || financial_year_end_day as date)
-            end
-        end as current_year_end_date,
-		source_relation
+        source_relation,
+        case when financial_year_end_month = 2 and financial_year_end_day = 29
+            then
+                case when cast({{ dbt.dateadd('day', -1, "cast(current_year || '-03-01' as date)") }} as date) >= current_date
+                    then cast({{ dbt.dateadd('day', -1, "cast(current_year || '-03-01' as date)") }} as date)
+                    else cast({{ dbt.dateadd('day', -1, "cast(next_year || '-03-01' as date)") }} as date)
+                    end
+            else
+                case when cast(current_year || '-' || financial_year_end_month || '-' || financial_year_end_day as date) >= current_date
+                    then cast(current_year || '-' || financial_year_end_month || '-' || financial_year_end_day as date)
+                    else cast(next_year || '-' || financial_year_end_month || '-' || financial_year_end_day as date)
+                    end
+        end as current_year_end_date
+
     from organization
 
 ), joined as (