Skip to content

Commit

Permalink
[Github #627] Nango Sync Beta (#667)
Browse files Browse the repository at this point in the history
* [gh-#627] initial groundwork

* [gh-#627] initial raw working version

* [gh-#627] incremental sync to only update or create if updated at is different

* [gh-#627] enum and add sync type

* [gh-#627] sync on a schedule in addition to the one time sync

* start logging

* [gh-#627] nangoConnectionId and add sync schedule table

* [gh-#627] start logic to use shared package

* [gh-#627] simplify connection service

* [gh-#632] add migration, service, and model

* [gh-#632] use db for storage and retrieval

* [gh-#632] oauth refactor

* [gh-#632] update models and methods

* [gh-#632] add sync type and remove commented out messages

* [gh-#627] bring in latest and db refactor

* [gh-#632] remove log compose file

* [gh-#627] move migrations and update methods for refactor

* [gh-#632] add index on account_id field

* [gh-#632] single query using json_agg

* [gh-#632] add cron deletion and notice about the deletion process

* [gh-#627] model movement

* [gh-#627] reduce number of dev commands to get started

• from root: `npm run dev:watch` `npm run dev:watch:apps`

* [gh-#627] remove dupe migrations

* [gh-#627] reorder migrations

* [gh-#627] fix capitalization

* [gh-#627] reorder packages

* [gh-#627] remove dupe method

* [gh-#627] remove dupe model

* [gh-#627] remove server from dep

* [gh-#627] add fe command and remove dupe activity service

* [gh-#627] reorder

* [gh-#627] data service implementation

* [gh-#627] return more data about the response

* [gh-#627] 1 hour sync

* [gh-#627] TODO note

* [gh-#637] poc working-ish version

* [gh-#637] remove webpack logic and write the js file

* [gh-#637] minor cleanup

* [gh-#637] docker update

* [gh-#637] add temporal dep

* [gh-#637] fix image name

* [gh-#637] add docker ignore

* [gh-#637] reference node client

* [gh-#637] add docker publish script

* [gh-#637] tighten up logs for syncs and make incrmental their own log entry

* [gh-#637] add sync server to be checked

* [gh-#637] update hq need

* [gh-#637] update image and dockerignore

* [gh-#637] update packages

* [gh-#637] update connection

* [gh-#637] remove buildinfo and update worker code

* [gh-#637] add asana quick working example

* [gh-#637] refactor yaml and upadte sync flow

* add sync_data_records with updated_at logic

* [gh-#637] move more to shared

* [gh-#637] add in getRecords method

* [gh-#637] move to dist directory and add simplified nango config object

* [gh-#637] rename sync-worker => worker

* [gh-#637] update schema reference

* [gh-#637] update script

• Usage should be `./scripts/docker-publish.bash nango-worker 0.1.0-rc`

* [gh-#637] hubspot api integration and start sync:run command

* [gh-#637] add lastSyncDate

* [gh-#637] fix container name

* [gh-#637] update container names

* [gh-#637] update nango-integrations

* [gh-#637] feat: Add better error logging when parsing of OAuth token response fails

* [gh-#637] fix output to dist file

* [gh-#637] use pass through to ensure the stream doesn't get cut off

* [gh-#637] Add Slack API base URL to providers.yaml

* [gh-#637] better error report handling and clean up

* [gh-#637] add docs

* [gh-#637] type generation logic and support multiple models

* [gh-#637] let, not var

* [gh-#637] update braintree dependency

* [gh-#637] update worker image

* [gh-#637] add ip package

* [gh-#637] bump version

* [gh-#637] watch for nango.yaml changes

* [gh-#637] don't overwrite file in the generate command

* [gh-#637] update interface types

* [gh-#637] remoe unused

* [gh-#637] decompress and ignoreInitial false

* [gh-#637] CLI sync:run: Pretty print nested JSON

* [gh-#637] add nullable types and undefined types

* [gh-#637] run check for any mjs files and make sure they're js

* [gh-#637] indentation

* [gh-#637] better logging and allow multiple models returned from a script to be inserted

* [gh-#637] add in params and paramsSerializer support

* [gh-#637] update uniqueness

* [gh-#637] onConflict with model

* [gh-#637] pass the model to get better result stats

* [gh-#637] dot env load for util

* [gh-#637] improve syncing logic

* Handle deduping
* Don't run the job in an iteration

* [gh-#637] bring in latest

* [gh-#637] sort logs correctly

* [gh-#637] sync status reflects better the result

* [gh-#637] delete schedule and logic fixes

* [gh-#637] A few more helpful log messages on sync creation

* [gh-#637] add offset and limit

* [gh-#637] use type guards

* [gh-#637] add dependency check workflow

* [gh-#637] path improvement

* [gh-#637] remove version

* [gh-#637] update usage

* [gh-#637] remove slash

* [gh-#637] beta release updates

* [gh-#637] bump server for webapp update

* [gh-#637] remove broken dep check

* [gh-#637] add example iteration code

* [gh-#637] add separate docker:run command

* [gh-#637] reload compiler on each change and compile all ts files on config file change

* [gh-#637] add field mapping logic

* [gh-#637] groundwork for ui sync page

* [gh-#637] refactor models and organization of syncs

* [gh-#637] ui sync tab clean up

* [gh-#637] sync ui polish and script for publishing

* [gh-#637] connection sync updates and batchSend cleanup

* [gh-#637] indentation

* [gh-#637] log improvements and internal linking from sync

* [gh-#637] sync page links back to a particular activity log

* [gh-#637] more cleanup and remove extra files

* [gh-#637] better alignment

* [gh-#637] spacing for longer models

* [gh-#637] log and triggerSync

* [gh-#637] version bump

* [gh-#637] add createConnection

* [gh-#637] activity log text wrapping

* [gh-#637] add import and update interval

* [gh-#637] add no referrer

* [gh-#637] Add airtable base API url

* [gh-#637] interval + offset implementation

* [gh-#637] tweak slightly

* [gh-#637] fix logic for interval calculation

* [gh-#637] getRecords returns data

* [gh-#637] lint fixes

* [gh-#637] webhook implementation

* [gh-#637] bump versions

* [gh-#637] update timestamp field and order of now

* [gh-#637] implement deploy to write to the db and use the nango config service to read

* [gh-#637] stub for file service

* [gh-#637] allow comma separated scopes to be entered

* [gh-#637] clean up

* [gh-#637] imported connection initiates sync

* [gh-#637] logging for the version

* [gh-#637] remove dupe environment

* [gh-#637] fix syncs that are days before forecast

* [gh-#637] add webhook type annotation

* [gh-#637] use vm2 to execute code

* [gh-#637] upload and download to/from s3

* [gh-#637] add release scripts

* [gh-#637] better handling if temporal is not running

* [gh-#637] upgrade integration template

* [gh-#637] activity status for an unfinished sync and env for script uploads

* [gh-#637] update activities logic

* [gh-#637] refactor and fix sync run

* [gh-#637] update sync run and add dry run logic

* log results
• add sync config to the job
• add job id to the data records

* [gh-#637] remove unused node-client sync

* [gh-#637] base url override logic

* [gh-#637] fix sync run

* [gh-#637] bump shared

* [gh-#637] fix token fetch showing on a dry sync run

* [gh-#637] allow emptying a webhook URL

* [gh-#637] empty unicode character to take up space

* [gh-#637] export github models

* [gh-#637] update file

* [gh-#637] safer delete call

* [gh-#637] remove unified_tickets table

* [gh-#637] clean up migration

* [gh-#637] bump versions

* [gh-#637] remove github models for now

* [gh-#637] use secrets for prod tls

* [gh-#637] update schedule if it changes

* [gh-#637] bump versions

* [gh-#637] update shared package version in cli

* [gh-#637] clean up script further

* [gh-#637] update script and publishing versions

* [gh-#637] version bumps

---------

Co-authored-by: Robin Guldener <[email protected]>
Co-authored-by: Bastien Beurier <[email protected]>
  • Loading branch information
3 people authored Jun 14, 2023
1 parent 43475c8 commit af698fe
Show file tree
Hide file tree
Showing 161 changed files with 15,373 additions and 8,615 deletions.
3 changes: 3 additions & 0 deletions .dockerignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
node_modules
**/node_modules
npm-debug.log
9 changes: 8 additions & 1 deletion .github/workflows/docker.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,14 @@ jobs:
CONTAINER_NAME=nango-db
DB=$(docker ps -q -f status=running -f name=^/${CONTAINER_NAME}$)
if [ ! "${DB}" ]; then
echo "DB doesn't exist"
echo "DB container doesn't exist"
exit 1
fi
CONTAINER_NAME=nango-worker
WORKER=$(docker ps -q -f status=running -f name=^/${CONTAINER_NAME}$)
if [ ! "${WORKER}" ]; then
echo "Worker container doesn't exist"
exit 1
fi
shell: bash
2 changes: 1 addition & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -4,4 +4,4 @@ logs/*
nango-data/*
.idea/*
.env

nango-integrations/*
35 changes: 33 additions & 2 deletions docker-compose.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -17,10 +17,11 @@ services:
nango-server:
image: nangohq/nango-server
container_name: nango-server
environment:
- TEMPORAL_ADDRESS=temporal:7233
- NANGO_DB_MIGRATION_FOLDER=/usr/nango-server/src/packages/shared/lib/db/migrations
volumes:
- './packages/shared/providers.yaml:/usr/nango-server/src/packages/shared/providers.yaml'
environment:
- NANGO_DB_MIGRATION_FOLDER=/usr/nango-server/src/packages/server/lib/db/migrations
restart: always
ports:
- '3003:3003'
Expand All @@ -29,5 +30,35 @@ services:
networks:
- nango

nango-worker:
image: nangohq/nango-worker
container_name: nango-worker
restart: always
ports:
- '3004:3004'
environment:
- TEMPORAL_ADDRESS=temporal:7233
depends_on:
- nango-db
- temporal
networks:
- nango

temporal:
image: temporalio/auto-setup
container_name: temporal
depends_on:
- nango-db
environment:
- DB=postgresql
- DB_PORT=5432
- POSTGRES_USER=nango
- POSTGRES_PWD=nango
- POSTGRES_SEEDS=nango-db
ports:
- 7233:7233
networks:
- nango

networks:
nango:
Loading

0 comments on commit af698fe

Please sign in to comment.