Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(data-ingestion): log end user #3086

Merged
merged 7 commits into from
Dec 3, 2024

Conversation

bodinsamuel
Copy link
Collaborator

@bodinsamuel bodinsamuel commented Nov 29, 2024

Changes

  • Log end user to BigQuery
  • Move endUser.service in shared because it's needed there
  • Try to have more typesafety in the schema
  • Try to add schema migration
  • Add feature flag so we can test locally and enable this feature more easily

@bodinsamuel bodinsamuel self-assigned this Nov 29, 2024
@bodinsamuel bodinsamuel marked this pull request as draft November 29, 2024 15:10
@bodinsamuel
Copy link
Collaborator Author

Tested in staging

@bodinsamuel bodinsamuel marked this pull request as ready for review December 2, 2024 09:09
@bodinsamuel bodinsamuel requested a review from a team December 2, 2024 09:09
Copy link
Contributor

@nalanj nalanj left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not able to run this because I don't have access to BigQuery (I don't think), but code looks reasonable. It feels like we have so much duplicated code under jobs/lib/execution, but I'm not really sure I can speak to it needing to be refactored or not.

Copy link
Collaborator

@TBonnin TBonnin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

haven't test the big query schema migration but the rest looks ok. Made a few comments but nothing blocking

const getEndUser = await getEndUserByConnectionId(db.knex, { connectionId: task.connection.id });
endUser = getEndUser.isOk()
? { id: getEndUser.value.id, endUserId: getEndUser.value.endUserId, orgId: getEndUser.value.organization?.organizationId || null }
: null;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

endUser is already null so why not if(getEndUser.isOk()) { endUser = ... }

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

indeed

@@ -158,3 +158,16 @@ export async function linkConnection(db: Knex, { endUserId, connection }: { endU
export async function unlinkConnection(db: Knex, { connection }: { connection: Pick<StoredConnection, 'id'> }) {
await db<StoredConnection>('_nango_connections').where({ id: connection.id! }).update({ end_user_id: null });
}

export async function getEndUserByConnectionId(db: Knex, props: { connectionId: number }): Promise<Result<EndUser, EndUserError>> {
const endUser = await db<DBEndUser>(END_USERS_TABLE)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we need the generic here since it is specified by the select below?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

true

@@ -392,6 +392,7 @@ export interface NangoProps {
runnerFlags: RunnerFlags;
debug: boolean;
startedAt: Date;
endUser: { id: number; endUserId: string | null; orgId: string | null } | null;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why not endUser: EndUser | null

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because the schema will change at some point and I don't need anything else, plus it's passing through zod which would be annoying to verify.
We do the same for team (and environment but differently)

@@ -392,6 +392,7 @@ export interface NangoProps {
runnerFlags: RunnerFlags;
debug: boolean;
startedAt: Date;
endUser: { id: number; endUserId: string | null; orgId: string | null } | null;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we passing the endUser only so we can have it when handling the job result. Have you thought about fetching it from the db when starting the jobs AND again when handling the job result?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, I'm not a fan. The way I see it, a job should be self-contained you send a payload and you receive the same thing, plus querying a new thing is one more possible way to fail. Tbf almost everything we put in NangoProps is not for execution per se.
Wdyt?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like the self-contained aspect of NangoProps but that's a lot of data doing the round trip. I am not sure what makes the most sense tbh

@bodinsamuel
Copy link
Collaborator Author

Im going to go ahead and merge since it's for billing, but happy to iterate if you find it useful ☺️ 👍🏻

@bodinsamuel bodinsamuel merged commit 32cd029 into master Dec 3, 2024
21 checks passed
@bodinsamuel bodinsamuel deleted the sam/24_11_29/fix/billing-end-user branch December 3, 2024 10:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants