Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

NEOS-1727, NEOS-1676, NEOS-1728: Adds docs for hooks, rbac, and more #3185

Draft
wants to merge 5 commits into
base: main
Choose a base branch
from
Draft

NEOS-1727, NEOS-1676, NEOS-1728: Adds docs for hooks, rbac, and more #3185

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
3ab3218
adds job hooks documentation
nickzelei Jan 23, 2025
63977ca
Adds docs page on rbac
nickzelei Jan 23, 2025
0d46bc4
adds missing column removal strategy docs page
nickzelei Jan 23, 2025
a647573
prettier
nickzelei Jan 23, 2025
7334889
Spellchecvk
nickzelei Jan 23, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 13 additions & 12 deletions docs/docs/deploy/environment-variables.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ description: A listing of the environment variables that Neosync uses to work lo
id: env-vars
hide_title: false
slug: /deploy/environment-variables
# cSpell:words PODCONFIG,APPNAME,LOKICONFIG,LABELSQUERY,KEEPLABELS,NEXTAUTH
# cSpell:words PODCONFIG,APPNAME,LOKICONFIG,LABELSQUERY,KEEPLABELS,NEXTAUTH,JOBHOOKS
---

## Overview
Expand Down Expand Up @@ -54,7 +54,7 @@ These environment variables are loaded when running the `mgmt serve connect` com
| KUBERNETES_NAMESPACE | Kubernetes namespace that API is running in | false | |
| KUBERNETES_WORKER_APP_NAME | App name of Worker running in Kubernetes | false | |
| METRICS_SERVICE_ENABLED | Whether or not to enable the metrics gRPC service | false | false |
| METRICS_URL | If the metrics service is enabled, this points it to the underlying prometheus instance | false | http://localhost:9090 |
| METRICS_URL | If the metrics service is enabled, this points it to the underlying prometheus instance | false | `http://localhost:9090` |
| METRICS_API_KEY | If the $METRICS_URL requires authentication, this will be passed to the api | false | |
| DB_LOG_LEVEL | Enables database query logging for postgres databases. This will also enable DB logging for traffic to the Neosync database. Note that if the $LOG_LEVEL is set to ERROR and $DB_LOG_LEVEL is set to INFO, you will only see ERROR level logs for the database as it uses the underlying logger. | false | NONE |
| LOG_LEVEL | Determines the level of logging that will be visible out of the backend logger. | false | INFO |
Expand Down Expand Up @@ -106,16 +106,17 @@ These environment variables are loaded when running the `mgmt migrate up` comman
| AUTH_PROVIDER_ID | The id of your provider. This is used by next-auth to identify the provider. This can really be anything you want. Required if AUTH_ENABLED is true. | false | |
| AUTH_PROVIDER_NAME | The name of your provider. This is used on the next-auth signin page and appears in the button text that a user will select. Required if AUTH_ENABLED is true. | false | |
| NEXT_PUBLIC_APP_BASE_URL | The url of the app. This is typically the same as NEXTAUTH_URL. Used for generating invite urls, among other things. This is not baked into the HTML or the image. | false | |
| NEOSYNC_API_BASE_URL | The base url of the Neosync API. This can be overridden to connect to different Neosync API environments | false | http://localhost:8080 |
| NEOSYNC_API_BASE_URL | The base url of the Neosync API. This can be overridden to connect to different Neosync API environments | false | `http://localhost:8080` |
| AUTH_AUTHORIZE_URL | The full url that routes to the authorize url. This is optional and may be necessary if you're using AUTH_EXPECTED_ISSUER as the urls will be different. This will need to include the public url (same value as AUTH_EXPECTED_ISSUER). If using a fully external auth provider, this will most likely not be needed. | false | |
| AUTH_USERINFO_URL | The full url that routes to the userinfo url. This is optional and may be necessary if you're using AUTH_EXPECTED_ISSUER as the urls will be different. This will most likely need to include the internal url (same value as AUTH_ISSUER). If using a fully external auth provider, this will most likely not be needed. | false | |
| AUTH_TOKEN_URL | The full url that routes to the token url. This is optional and may be necessary if you're using AUTH_EXPECTED_ISSUER as the urls will be different. This will most likely need to include the internal url (same value as AUTH_ISSUER). If using a fully external auth provider, this will most likely not be needed. | false | |
| AUTH_TRUST_HOST | Whether or not to trust the external host. Check the next-auth docs for more details. This is pretty must required if running behind a load balancer. | true | |
| NEOSYNC_ANALYTICS_ENABLED | Enables analytics within the Neosync app. Disable to turn off all analytics. | false | true |
| POSTHOG_KEY | API key used to send analytics to Posthog. Without this, analytics will be disabled, even if `NEOSYNC_ANALYTICS_ENABLED` is set to true. | false | Neosync's posthog api key |
| POSTHOG_HOST | Used to change the host of the Posthog instance that analytics are shipped to. Only needed if self-hosting Posthog or not shipping to the default Posthog platform. | false | https://app.posthog.com |
| POSTHOG_HOST | Used to change the host of the Posthog instance that analytics are shipped to. Only needed if self-hosting Posthog or not shipping to the default Posthog platform. | false | `https://app.posthog.com` |
| ENABLE_RUN_LOGS | Used to enable job run logs on job run page. Worker and API must be running in Kubernetes | false | false |
| METRICS_SERVICE_ENABLED | Whether or not to enable displaying metrics. This may only be enabled if the backend also has the metrics service configured | false | false |
| JOBHOOKS_ENABLED | Whether or not to surface job hooks in the app. Note, in order for the job hooks service to be available on the backend, a valid Neosync EE License is required. | false | |

## Worker

Expand Down Expand Up @@ -153,14 +154,14 @@ These environment variables are loaded when running the `worker serve` command w

There are some environment variables that the CLI accepts to override default behavior to accommodate different environments.

| Variable | Description | Is Required | Default Value |
| ------------------- | ------------------------------------------------------------------------------------------------------------------ | ----------- | --------------------- |
| NEOSYNC_API_URL | The base url of the Neosync API. This can be overridden to connect to different Neosync API environments | false | http://localhost:8080 |
| NEOSYNC_API_KEY | The api key for Neosync API. | false | |
| LOGIN_HOST | The http server that is booted up running `neosync login` via an oauth flow | false | 127.0.0.1 |
| LOGIN_REDIRECT_HOST | The redirect host that is sent alongside the oauth flow when running `neosync login` | false | 127.0.0.1 |
| LOGIN_PORT | The port the http server runs on when running `neosync login` | false | 4242 |
| NEOSYNC_CONFIG_DIR | The config directory to store Neosync-specific credentials. For Linux users, `$XDG_CONFIG_HOME` is also respected. | false | ~/.neosync |
| Variable | Description | Is Required | Default Value |
| ------------------- | ------------------------------------------------------------------------------------------------------------------ | ----------- | ----------------------- |
| NEOSYNC_API_URL | The base url of the Neosync API. This can be overridden to connect to different Neosync API environments | false | `http://localhost:8080` |
| NEOSYNC_API_KEY | The api key for Neosync API. | false | |
| LOGIN_HOST | The http server that is booted up running `neosync login` via an oauth flow | false | 127.0.0.1 |
| LOGIN_REDIRECT_HOST | The redirect host that is sent alongside the oauth flow when running `neosync login` | false | 127.0.0.1 |
| LOGIN_PORT | The port the http server runs on when running `neosync login` | false | 4242 |
| NEOSYNC_CONFIG_DIR | The config directory to store Neosync-specific credentials. For Linux users, `$XDG_CONFIG_HOME` is also respected. | false | ~/.neosync |

## Open Telemetry

Expand Down
2 changes: 1 addition & 1 deletion docs/docs/guides/ai-synthetic-data-generation.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ hide_title: false
slug: /guides/ai-synthetic-data-generation
---

# Intro
## Intro

Now you can use any model that is hosted on an endpoint to be able to generate synthetic data.

Expand Down
35 changes: 35 additions & 0 deletions docs/docs/guides/column-removal-strategies.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
---
title: Column Removal Strategies
description: Learn how to configure Neosync to handle old columns that no longer exist in the source database
id: column-removal-strategies
hide_title: false
slug: /guides/column-removal-strategies
---

## Introduction

When a Neosync Job is configured for relational databases, all columns for each selected table must have a transformer mapping configured.
This page goes into detail how Neosync handles old columns that no longer exist in the source database and the different strategies that can be used to handle this.

This is a common occurrence for any company that is adding or removing columns to a database and may not update Neosync straight away.

The `continue` strategy is the default strategy as it is the most flexible. The idea is to keep Neosync running and being less brittle to configuration drift without having to constantly check in on how Neosync is doing.

## Driver Support

| Strategy | Description | PostgreSQL | MySQL | MS SQL Server |
| -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- | ----- | ------------- |
| Halt | Stops the job run if a column is configured in the job mappings but no longer exists in the source database. | ✅ | ✅ | ✅ |
| Continue | Columns are ignored from the source and are not inserted into the destination; This may fail if column still exists in the destination but does not have a default value configured. See more below. | ✅ | ✅ | ✅ |

## Halt Strategy

This strategy is plain and simple. During the job run, Neosync compares the configured job mappings with the source database.
For the selected tables in the job mappings, a diff is made and if a column is found in the job mappings that doesn't exist in the source database, the run is halted.

## Continue Strategy

This strategy tells Neosync to ignore any difference in job mappings from the source database.

Neosync is able to detect that columns were removed in the source, and will leave them off of the insert statement.
This may result in failures if any unmapped columns do not have a column default in the destination connection.
28 changes: 22 additions & 6 deletions docs/docs/guides/initializing-schemas.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,13 +6,29 @@ hide_title: false
slug: /guides/initializing-schemas
---

## Background
## Introduction

Neosync has a feature that will "Initialize" your schema. However this feature is not always the best way to get your schema set up. This guide will walk you through ways to set up your schema manually.
Neosync has the ability to initialize your database schema prior to running a sync. This is useful if you have a lot of migrations or if you want to ensure that your schema is exactly the same as your source.

## Option 1: Run your Migrations
## Driver Support

If you have a set of migrations that you use to set up your database, you can run these migrations on your target database. This is the most common way to set up your schema.
Postgres, MySQL, and MS SQL Server are supported.

> MS SQL Server requires an Enterprise License for OSS users.

## Option 1: Use Neosync's Initialize Schema Feature

The simplest way to get started is to simply enable the feature in your job's configuration.

Once this is enabled, Neosync will generate SQL statements based on the schemas and tables you've selected in your job's configuration to be created prior to syncing the data itself.

Neosync tries to be as smart as possible about what to include in the schema, but it is not always perfect. If you find that you are missing something, you can always manually add it to the schema, or reach out to us to help you out.

> Some actions like schema creation, extension installation, Sequence creation and resets require schema owner access.

## Option 2: Run your Migrations

If you have a set of migrations that you use to set up your database, you can run these migrations on your target database. This is a common way to set up your schema.

This can potentially have some issues if you have a lot of migrations or if your migrations are not idempotent.

Expand All @@ -26,9 +42,9 @@ Some of the above issues can be resolved by choosing to run your migrations agai

Steps to run this method are out of scope for the Neosync docs, but ideally it is as simple as running your migrations against the same database as the Neosync destination.

## Option 2: `pg_dump` your Schema
## Option 3: `pg_dump` your schema (or other driver equivalents)

Option 2 is to use `pg_dump` to dump your schema and then apply it manually to your target database.
Use `pg_dump` to dump your schema and then apply it manually to your target database.

This has a few advantages over the above method:

Expand Down
Loading
Loading