Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Updates job mapping validator to return error codes #3179

Merged
merged 9 commits into from
Jan 23, 2025

Conversation

alishakawaguchi
Copy link
Contributor

No description provided.

@alishakawaguchi alishakawaguchi added the enhancement New feature or request label Jan 22, 2025
@alishakawaguchi alishakawaguchi self-assigned this Jan 22, 2025
Copy link

linear bot commented Jan 22, 2025

Copy link

vercel bot commented Jan 22, 2025

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name Status Preview Comments Updated (UTC)
neosync-docs ✅ Ready (Inspect) Visit Preview 💬 Add feedback Jan 23, 2025 6:42am

Copy link

github-actions bot commented Jan 22, 2025

The latest Buf updates on your PR. Results from workflow Buf / buf (pull_request).

BuildFormatLintBreakingUpdated (UTC)
✅ passed✅ passed✅ passed✅ passedJan 23, 2025, 6:42 AM

Copy link

github-actions bot commented Jan 22, 2025

Benchstat Geomean Results

0.43% sec/op, -0.03% B/op, 0.00% allocs/op

Benchstat results
                                   │ main-benchmark.txt │        branch-benchmark.txt         │
                                   │       sec/op       │   sec/op     vs base                │
CleanPostgresType-4                        41.12n ± ∞ ¹   42.67n ± 1%       ~ (p=0.286 n=1+6)
Runner_Single-4                            7.942µ ± ∞ ¹   7.819µ ± 7%       ~ (p=0.571 n=1+6)
GenerateBool-4                             11.76n ± ∞ ¹   11.77n ± 4%       ~ (p=1.000 n=1+6)
GenerateBusinessName-4                     334.8n ± ∞ ¹   321.8n ± 4%       ~ (p=0.286 n=1+6)
GenerateCardNumber-4                       28.64n ± ∞ ¹   28.60n ± 0%       ~ (p=0.571 n=1+6)
GenerateCategorical-4                      102.3n ± ∞ ¹   103.3n ± 1%       ~ (p=0.571 n=1+6)
GenerateCity-4                             173.1n ± ∞ ¹   175.2n ± 1%       ~ (p=0.286 n=1+6)
GenerateCountry-4                          105.2n ± ∞ ¹   106.0n ± 1%       ~ (p=0.286 n=1+6)
GenerateEmail-4                            1.383µ ± ∞ ¹   1.356µ ± 1%       ~ (p=0.286 n=1+6)
GenerateFirstName-4                        236.7n ± ∞ ¹   209.6n ± 2%       ~ (p=0.286 n=1+6)
GenerateFloat64-4                          28.72n ± ∞ ¹   28.97n ± 1%       ~ (p=0.286 n=1+6)
GenerateFullAddress-4                      1.430µ ± ∞ ¹   1.448µ ± 1%       ~ (p=0.286 n=1+6)
GenerateFullName-4                         1.299µ ± ∞ ¹   1.427µ ± 2%       ~ (p=0.286 n=1+6)
GenerateGender-4                           41.98n ± ∞ ¹   41.83n ± 0%  -0.36% (n=1+6)
GenerateInt64-4                            28.00n ± ∞ ¹   28.41n ± 0%       ~ (p=0.286 n=1+6)
GenerateInt64PhoneNumber-4                 51.76n ± ∞ ¹   51.70n ± 0%       ~ (p=0.857 n=1+6)
GenerateInternationalPhoneNumber-4         176.1n ± ∞ ¹   176.1n ± 1%       ~ (p=1.000 n=1+6)
GenerateIpAddress-4                        568.1n ± ∞ ¹   544.8n ± 1%       ~ (p=0.286 n=1+6)
GenerateLastName-4                         196.7n ± ∞ ¹   200.8n ± 2%       ~ (p=0.286 n=1+6)
GenerateRandomString-4                    1007.0n ± ∞ ¹   989.0n ± 2%       ~ (p=0.571 n=1+6)
GenerateSHA256Hash-4                       1.007µ ± ∞ ¹   1.016µ ± 0%       ~ (p=0.286 n=1+6)
GenerateSSN-4                              301.8n ± ∞ ¹   303.5n ± 0%       ~ (p=0.286 n=1+6)
GenerateState-4                            104.9n ± ∞ ¹   106.0n ± 1%       ~ (p=0.286 n=1+6)
GenerateStreetAddress-4                    205.3n ± ∞ ¹   217.3n ± 1%       ~ (p=0.286 n=1+6)
GenerateStringPhoneNumber-4                173.9n ± ∞ ¹   179.4n ± 3%       ~ (p=0.286 n=1+6)
GenerateUnixTimestamp-4                    81.14n ± ∞ ¹   77.93n ± 3%       ~ (p=0.286 n=1+6)
GenerateUsername-4                         354.3n ± ∞ ¹   386.7n ± 5%       ~ (p=0.286 n=1+6)
GenerateUTCTimestamp-4                     107.1n ± ∞ ¹   104.2n ± 0%       ~ (p=0.286 n=1+6)
GenerateUUID-4                             738.8n ± ∞ ¹   734.5n ± 1%       ~ (p=0.286 n=1+6)
GenerateZipcode-4                          105.5n ± ∞ ¹   106.0n ± 0%       ~ (p=0.286 n=1+6)
TransformCharacterScramble-4               173.7n ± ∞ ¹   172.6n ± 1%       ~ (p=0.286 n=1+6)
TransformE164PhoneNumber-4                 180.9n ± ∞ ¹   181.7n ± 0%       ~ (p=0.286 n=1+6)
TransformEmail-4                           1.791µ ± ∞ ¹   1.822µ ± 0%       ~ (p=0.286 n=1+6)
TransformFirstName-4                       239.7n ± ∞ ¹   238.7n ± 2%       ~ (p=0.571 n=1+6)
TransformFloat64-4                         111.5n ± ∞ ¹   116.8n ± 1%       ~ (p=0.286 n=1+6)
TransformFullName-4                        1.304µ ± ∞ ¹   1.306µ ± 1%       ~ (p=0.571 n=1+6)
TransformInt64-4                           33.45n ± ∞ ¹   33.68n ± 1%       ~ (p=0.286 n=1+6)
TransformInt64PhoneNumber-4                58.74n ± ∞ ¹   58.03n ± 0%       ~ (p=0.286 n=1+6)
TransformLastName-4                        197.3n ± ∞ ¹   198.8n ± 1%       ~ (p=0.286 n=1+6)
TransformString-4                          1.026µ ± ∞ ¹   1.028µ ± 1%       ~ (p=0.857 n=1+6)
TransformStringPhoneNumber-4               205.1n ± ∞ ¹   210.6n ± 0%       ~ (p=0.286 n=1+6)
TransformUuid-4                            36.66n ± ∞ ¹   36.80n ± 1%       ~ (p=0.286 n=1+6)
geomean                                    197.0n         197.8n       +0.43%
¹ need >= 6 samples for confidence interval at level 0.95

                                   │ main-benchmark.txt │          branch-benchmark.txt          │
                                   │        B/op        │     B/op      vs base                  │
CleanPostgresType-4                         0.000 ± ∞ ¹     0.000 ± 0%       ~ (p=1.000 n=1+6) ²
Runner_Single-4                           2.318Ki ± ∞ ¹   2.290Ki ± 1%       ~ (p=0.286 n=1+6)
GenerateBool-4                              0.000 ± ∞ ¹     0.000 ± 0%       ~ (p=1.000 n=1+6) ²
GenerateBusinessName-4                      304.0 ± ∞ ¹     304.0 ± 0%       ~ (p=1.000 n=1+6) ²
GenerateCardNumber-4                        8.000 ± ∞ ¹     8.000 ± 0%       ~ (p=1.000 n=1+6) ²
GenerateCategorical-4                       64.00 ± ∞ ¹     64.00 ± 0%       ~ (p=1.000 n=1+6) ²
GenerateCity-4                              160.0 ± ∞ ¹     160.0 ± 0%       ~ (p=1.000 n=1+6) ²
GenerateCountry-4                           24.00 ± ∞ ¹     24.00 ± 0%       ~ (p=1.000 n=1+6) ²
GenerateEmail-4                             385.0 ± ∞ ¹     385.0 ± 0%       ~ (p=1.000 n=1+6) ²
GenerateFirstName-4                         208.0 ± ∞ ¹     208.0 ± 0%       ~ (p=1.000 n=1+6) ²
GenerateFloat64-4                           8.000 ± ∞ ¹     8.000 ± 0%       ~ (p=1.000 n=1+6) ²
GenerateFullAddress-4                       512.0 ± ∞ ¹     512.0 ± 0%       ~ (p=1.000 n=1+6) ²
GenerateFullName-4                          418.0 ± ∞ ¹     418.0 ± 0%       ~ (p=1.000 n=1+6) ²
GenerateGender-4                            16.00 ± ∞ ¹     16.00 ± 0%       ~ (p=1.000 n=1+6) ²
GenerateInt64-4                             7.000 ± ∞ ¹     7.000 ± 0%       ~ (p=1.000 n=1+6) ²
GenerateInt64PhoneNumber-4                  8.000 ± ∞ ¹     8.000 ± 0%       ~ (p=1.000 n=1+6) ²
GenerateInternationalPhoneNumber-4          40.00 ± ∞ ¹     40.00 ± 0%       ~ (p=1.000 n=1+6) ²
GenerateIpAddress-4                         31.00 ± ∞ ¹     31.00 ± 0%       ~ (p=1.000 n=1+6) ²
GenerateLastName-4                          176.0 ± ∞ ¹     176.0 ± 0%       ~ (p=1.000 n=1+6) ²
GenerateRandomString-4                      186.0 ± ∞ ¹     186.0 ± 0%       ~ (p=1.000 n=1+6) ²
GenerateSHA256Hash-4                        288.0 ± ∞ ¹     288.0 ± 0%       ~ (p=1.000 n=1+6) ²
GenerateSSN-4                               46.00 ± ∞ ¹     46.00 ± 0%       ~ (p=1.000 n=1+6) ²
GenerateState-4                             24.00 ± ∞ ¹     24.00 ± 0%       ~ (p=1.000 n=1+6) ²
GenerateStreetAddress-4                     240.0 ± ∞ ¹     240.0 ± 0%       ~ (p=1.000 n=1+6) ²
GenerateStringPhoneNumber-4                 40.00 ± ∞ ¹     40.00 ± 0%       ~ (p=1.000 n=1+6) ²
GenerateUnixTimestamp-4                     8.000 ± ∞ ¹     8.000 ± 0%       ~ (p=1.000 n=1+6) ²
GenerateUsername-4                          205.0 ± ∞ ¹     205.0 ± 0%       ~ (p=1.000 n=1+6) ²
GenerateUTCTimestamp-4                      24.00 ± ∞ ¹     24.00 ± 0%       ~ (p=1.000 n=1+6) ²
GenerateUUID-4                              80.00 ± ∞ ¹     80.00 ± 0%       ~ (p=1.000 n=1+6) ²
GenerateZipcode-4                           24.00 ± ∞ ¹     24.00 ± 0%       ~ (p=1.000 n=1+6) ²
TransformCharacterScramble-4                40.00 ± ∞ ¹     40.00 ± 3%       ~ (p=0.286 n=1+6)
TransformE164PhoneNumber-4                  40.00 ± ∞ ¹     40.00 ± 0%       ~ (p=1.000 n=1+6) ²
TransformEmail-4                            489.0 ± ∞ ¹     489.0 ± 0%       ~ (p=1.000 n=1+6) ²
TransformFirstName-4                        224.0 ± ∞ ¹     224.0 ± 0%       ~ (p=1.000 n=1+6) ²
TransformFloat64-4                          88.00 ± ∞ ¹     88.00 ± 0%       ~ (p=1.000 n=1+6) ²
TransformFullName-4                         418.0 ± ∞ ¹     418.0 ± 0%       ~ (p=1.000 n=1+6) ²
TransformInt64-4                            8.000 ± ∞ ¹     8.000 ± 0%       ~ (p=1.000 n=1+6) ²
TransformInt64PhoneNumber-4                 8.000 ± ∞ ¹     8.000 ± 0%       ~ (p=1.000 n=1+6) ²
TransformLastName-4                         176.0 ± ∞ ¹     176.0 ± 0%       ~ (p=1.000 n=1+6) ²
TransformString-4                           202.0 ± ∞ ¹     202.0 ± 0%       ~ (p=1.000 n=1+6) ²
TransformStringPhoneNumber-4                56.00 ± ∞ ¹     56.00 ± 0%       ~ (p=1.000 n=1+6) ²
TransformUuid-4                             16.00 ± ∞ ¹     16.00 ± 0%       ~ (p=1.000 n=1+6) ²
geomean                                               ³                 -0.03%                 ³
¹ need >= 6 samples for confidence interval at level 0.95
² all samples are equal
³ summaries must be >0 to compute geomean

                                   │ main-benchmark.txt │         branch-benchmark.txt         │
                                   │     allocs/op      │ allocs/op   vs base                  │
CleanPostgresType-4                         0.000 ± ∞ ¹   0.000 ± 0%       ~ (p=1.000 n=1+6) ²
Runner_Single-4                             24.00 ± ∞ ¹   24.00 ± 0%       ~ (p=1.000 n=1+6) ²
GenerateBool-4                              0.000 ± ∞ ¹   0.000 ± 0%       ~ (p=1.000 n=1+6) ²
GenerateBusinessName-4                      2.000 ± ∞ ¹   2.000 ± 0%       ~ (p=1.000 n=1+6) ²
GenerateCardNumber-4                        1.000 ± ∞ ¹   1.000 ± 0%       ~ (p=1.000 n=1+6) ²
GenerateCategorical-4                       2.000 ± ∞ ¹   2.000 ± 0%       ~ (p=1.000 n=1+6) ²
GenerateCity-4                              2.000 ± ∞ ¹   2.000 ± 0%       ~ (p=1.000 n=1+6) ²
GenerateCountry-4                           2.000 ± ∞ ¹   2.000 ± 0%       ~ (p=1.000 n=1+6) ²
GenerateEmail-4                             8.000 ± ∞ ¹   8.000 ± 0%       ~ (p=1.000 n=1+6) ²
GenerateFirstName-4                         2.000 ± ∞ ¹   2.000 ± 0%       ~ (p=1.000 n=1+6) ²
GenerateFloat64-4                           1.000 ± ∞ ¹   1.000 ± 0%       ~ (p=1.000 n=1+6) ²
GenerateFullAddress-4                       10.00 ± ∞ ¹   10.00 ± 0%       ~ (p=1.000 n=1+6) ²
GenerateFullName-4                          6.000 ± ∞ ¹   6.000 ± 0%       ~ (p=1.000 n=1+6) ²
GenerateGender-4                            1.000 ± ∞ ¹   1.000 ± 0%       ~ (p=1.000 n=1+6) ²
GenerateInt64-4                             0.000 ± ∞ ¹   0.000 ± 0%       ~ (p=1.000 n=1+6) ²
GenerateInt64PhoneNumber-4                  1.000 ± ∞ ¹   1.000 ± 0%       ~ (p=1.000 n=1+6) ²
GenerateInternationalPhoneNumber-4          3.000 ± ∞ ¹   3.000 ± 0%       ~ (p=1.000 n=1+6) ²
GenerateIpAddress-4                         2.000 ± ∞ ¹   2.000 ± 0%       ~ (p=1.000 n=1+6) ²
GenerateLastName-4                          2.000 ± ∞ ¹   2.000 ± 0%       ~ (p=1.000 n=1+6) ²
GenerateRandomString-4                      3.000 ± ∞ ¹   3.000 ± 0%       ~ (p=1.000 n=1+6) ²
GenerateSHA256Hash-4                        7.000 ± ∞ ¹   7.000 ± 0%       ~ (p=1.000 n=1+6) ²
GenerateSSN-4                               3.000 ± ∞ ¹   3.000 ± 0%       ~ (p=1.000 n=1+6) ²
GenerateState-4                             2.000 ± ∞ ¹   2.000 ± 0%       ~ (p=1.000 n=1+6) ²
GenerateStreetAddress-4                     2.000 ± ∞ ¹   2.000 ± 0%       ~ (p=1.000 n=1+6) ²
GenerateStringPhoneNumber-4                 3.000 ± ∞ ¹   3.000 ± 0%       ~ (p=1.000 n=1+6) ²
GenerateUnixTimestamp-4                     1.000 ± ∞ ¹   1.000 ± 0%       ~ (p=1.000 n=1+6) ²
GenerateUsername-4                          6.000 ± ∞ ¹   6.000 ± 0%       ~ (p=1.000 n=1+6) ²
GenerateUTCTimestamp-4                      1.000 ± ∞ ¹   1.000 ± 0%       ~ (p=1.000 n=1+6) ²
GenerateUUID-4                              3.000 ± ∞ ¹   3.000 ± 0%       ~ (p=1.000 n=1+6) ²
GenerateZipcode-4                           2.000 ± ∞ ¹   2.000 ± 0%       ~ (p=1.000 n=1+6) ²
TransformCharacterScramble-4                2.000 ± ∞ ¹   2.000 ± 0%       ~ (p=1.000 n=1+6) ²
TransformE164PhoneNumber-4                  3.000 ± ∞ ¹   3.000 ± 0%       ~ (p=1.000 n=1+6) ²
TransformEmail-4                            14.00 ± ∞ ¹   14.00 ± 0%       ~ (p=1.000 n=1+6) ²
TransformFirstName-4                        3.000 ± ∞ ¹   3.000 ± 0%       ~ (p=1.000 n=1+6) ²
TransformFloat64-4                          3.000 ± ∞ ¹   3.000 ± 0%       ~ (p=1.000 n=1+6) ²
TransformFullName-4                         6.000 ± ∞ ¹   6.000 ± 0%       ~ (p=1.000 n=1+6) ²
TransformInt64-4                            1.000 ± ∞ ¹   1.000 ± 0%       ~ (p=1.000 n=1+6) ²
TransformInt64PhoneNumber-4                 1.000 ± ∞ ¹   1.000 ± 0%       ~ (p=1.000 n=1+6) ²
TransformLastName-4                         2.000 ± ∞ ¹   2.000 ± 0%       ~ (p=1.000 n=1+6) ²
TransformString-4                           4.000 ± ∞ ¹   4.000 ± 0%       ~ (p=1.000 n=1+6) ²
TransformStringPhoneNumber-4                4.000 ± ∞ ¹   4.000 ± 0%       ~ (p=1.000 n=1+6) ²
TransformUuid-4                             1.000 ± ∞ ¹   1.000 ± 0%       ~ (p=1.000 n=1+6) ²
geomean                                               ³               +0.00%                 ³
¹ need >= 6 samples for confidence interval at level 0.95
² all samples are equal
³ summaries must be >0 to compute geomean

Copy link

codecov bot commented Jan 22, 2025

Codecov Report

Attention: Patch coverage is 60.00000% with 38 lines in your changes missing coverage. Please review.

Project coverage is 27.85%. Comparing base (982a282) to head (9e3bc93).
Report is 2 commits behind head on main.

Files with missing lines Patch % Lines
backend/services/mgmt/v1alpha1/job-service/jobs.go 0.00% 38 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #3179      +/-   ##
==========================================
+ Coverage   27.82%   27.85%   +0.02%     
==========================================
  Files         378      378              
  Lines       43259    43307      +48     
==========================================
+ Hits        12037    12061      +24     
- Misses      29772    29797      +25     
+ Partials     1450     1449       -1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

message ColumnError {
// The schema of the table
string schema = 1;
// The table of the column
string table = 2;
// The column of the error
string column = 3;
// The list of errors
// @deprecated - Use error_reports instead
repeated string errors = 4;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

also worthwhile to add the [deprecated = true] rpc option to anywhere that you are using @deprecated

type JobMappingErrorCode string

const (
TABLE_NOT_FOUND_IN_SOURCE JobMappingErrorCode = "TABLE_NOT_FOUND_IN_SOURCE"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

these are great. so descriptive.

repeated string errors = 4;
// The list of error reports
repeated ErrorReport error_reports = 5;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we could potentially be more descriptive here and move the enum you described out of Go and into the proto layer so that you have high quality error code typings in the frontend.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, this is interesting, but does it make sense to have the ErrorReport specific to ColumnError ColumnWarning and DatabaseError? From there you could describe even further scoped error codes that are relevant only to those? Maybe let's talk more about this if this makes sense.

@alishakawaguchi alishakawaguchi requested review from nickzelei and removed request for nickzelei January 22, 2025 23:36
@alishakawaguchi alishakawaguchi merged commit cfeb6fb into main Jan 23, 2025
22 checks passed
@alishakawaguchi alishakawaguchi deleted the alisha/neos-1725-error-codes branch January 23, 2025 18:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants