Feature branch: PDS v2 (bluesky-social#1789)

* cleanup repeat process all * wip * skip actor search test * skip actor search test * tweak processAll * decrease wait to 1 sec * repo_blob -> record_blob * simplify backlink linkTo * return repo_root to one row * sequence before updating repo_root * invite code forUser -> forAccount * ipld_block -> repo_block * use lru-cache fetchMethod * move did_cache to own db * better error handling on did cache * drop did_handle * fix sequencer wait time * debug * debug * more debug * check something * fix bday paradox * fix bday paradox * tidy up pds service auth * rm skipped test * retry http * tidy * improve fanout error handling * fix test * return signing key in did-web * more tests * tidy serivce auth checks * user_account -> account * remove inviteNote * keypair per repo * use an lru cache for keypairs as well * clean up repo * wip * wrap up accoutn manager * tidy * tidy * fix tests * fix disabled codes * fix appview tests * add note * set pragmas * tidy account manager getDb * rename pref transactor * user pref -> account pref * handle blob imports * tidy imports * add reserveSigningKey * wip transferAccount * clean up transferAccount * tests * tidy * tidy * configure entryway url on pds * handle entryway in pds admin endpoints * make importRepo temp * fix imports * make email optional on pds when using entryway * handle diffs * handle pds entryway usage for server, identity, admin endpoints * pds support for credentials from entryway * setup pds tests w/ entryway service * tidy * tidy * update entryway version * wip * test handle updates w/ entryway * split account table into two * tidy * tweak scripts * tidy tests * tidy * better config for actorstore & dbs * clean up cfg more * reorg actorstore fs layout * handle erros on actor db create * pr tidy & fix accoutn deletion test * pr feedback * fix bad merge * unskip test * fix subscribe repos tests * tidy repo root tables * tidy * fix tests * tidy delete tokens * tidy account getters * tidy * bulk deletesg * increase chunk size * handle racing refreshes * wip * fix auth test * invert import flow * clean up actor store on create account failure * tweak sequencer * prevent invite code races on createAccount * rm note * add back in race protection on getAccountInviteCodes * start feature branch * deleted app migration table * patch up new auth test * rm note * g * create accoutn delegated from entryway * tidy * fix test * change plcOp type to unknown * small fixes * sync up w entryway branch * Use proper error when authed account is not found (bluesky-social#1799) provide proper error when account not found in access-takedown check * build branch * build on ghcr * tweak service file * tweak service file * change where we save reserved keys * no tmp dir in blobstore either * fix blobstore temp location again * handle repeat record_blobs * create account before submitting plc op & undo if fail * small tweak * limit the number of local records * push out empty commit on transfer * fix issue with record_blob * add push blob endpoint * Set and validate token audiences on pds v2 (bluesky-social#1793) set and validate token audience on pds v2 * merge * include entryway did on tests * build branch * fix cache issue * xrpc server blob limit * put correct bytes * add auth to routes * handle quarantining/unquarantining a blob that does not exist * tidy * fix transfer tests * fix email request routes for entryway * PDS v2 entryway account deletion (bluesky-social#1819) * add admin lexicon for account deletion * implement admin account deletion endpoint * fix entryway proxying on account email checks * proxy to entryway for acct deletion * read-after-write sanity check * tweak * wip * finish refactor * fix test schema * application retry logic for busy * pr feedback * rm lru-cache * fix test pg schema * fix transfer test * Sqlite instrumentation for pds v2 (bluesky-social#1838) * sqlite instrumentation * build * remove build * dont reimport blobs * send ticks during import * close on error * catch handle validation error * add log * fix test * return emailConfirmedAt on getAccountInfo * Upgrade sharp on pds v2 (bluesky-social#1863) upgrade sharp to 0.32.6 * read all bytes before parsing car * Async car reader (bluesky-social#1867) * asynchronously read in car * dont buffer car * tweak * Gracefully handle indexing of invalid records (bluesky-social#1853) * gracefully handle indexing of invalid records * fix repo tests * Fix role auth for access-or-role verifier, getBlob check on actor takedowns (bluesky-social#1869) fix role auth for access-or-role verifier, fix getBlob actor takedown check * better cleanup of actor-stores * add ability to not ensure leaves * tidy * allow did:web transfer * Migration utility for actor-store (bluesky-social#1873) beginnings of helper for migrating all actors Co-authored-by: Devin Ivy <[email protected]> * base case for findBlobRefs * App-level retries for sqlite on pds (bluesky-social#1871) * revamp retry helper to be more flexible re: backoff strategies * sqlite timeout helper * ensure sqlite wal on db creation/migration rather than every open * layer retries for sqlite on writes outside transactions on pds * tidy * fix up lockfile * tidy * fix lex codegen * fix timing bug in threadgate test * No-op update handling (bluesky-social#1916) do no produce commits on no-op updates * Retry on all SQLITE_BUSY error codes (bluesky-social#1917) retry on all sqlite_busy error codes * Pds v2 ensure sqlite ready (bluesky-social#1918) ensure sqlite is ready before making queries * try something * tidy * dont build branch --------- Co-authored-by: Devin Ivy <[email protected]>
kemerald25 · Dec 5, 2023 · f9fd3e6 · f9fd3e6
1 parent cad30a7
commit f9fd3e6
Show file tree

Hide file tree

Showing 278 changed files with 8,597 additions and 6,920 deletions.
diff --git a/lexicons/com/atproto/admin/deleteAccount.json b/lexicons/com/atproto/admin/deleteAccount.json
@@ -0,0 +1,20 @@
+{
+  "lexicon": 1,
+  "id": "com.atproto.admin.deleteAccount",
+  "defs": {
+    "main": {
+      "type": "procedure",
+      "description": "Delete a user account as an administrator.",
+      "input": {
+        "encoding": "application/json",
+        "schema": {
+          "type": "object",
+          "required": ["did"],
+          "properties": {
+            "did": { "type": "string", "format": "did" }
+          }
+        }
+      }
+    }
+  }
+}
diff --git a/lexicons/com/atproto/temp/importRepo.json b/lexicons/com/atproto/temp/importRepo.json
@@ -0,0 +1,27 @@
+{
+  "lexicon": 1,
+  "id": "com.atproto.temp.importRepo",
+  "defs": {
+    "main": {
+      "type": "procedure",
+      "description": "Gets the did's repo, optionally catching up from a specific revision.",
+      "parameters": {
+        "type": "params",
+        "required": ["did"],
+        "properties": {
+          "did": {
+            "type": "string",
+            "format": "did",
+            "description": "The DID of the repo."
+          }
+        }
+      },
+      "input": {
+        "encoding": "application/vnd.ipld.car"
+      },
+      "output": {
+        "encoding": "text/plain"
+      }
+    }
+  }
+}
diff --git a/lexicons/com/atproto/temp/pushBlob.json b/lexicons/com/atproto/temp/pushBlob.json
@@ -0,0 +1,24 @@
+{
+  "lexicon": 1,
+  "id": "com.atproto.temp.pushBlob",
+  "defs": {
+    "main": {
+      "type": "procedure",
+      "description": "Gets the did's repo, optionally catching up from a specific revision.",
+      "parameters": {
+        "type": "params",
+        "required": ["did"],
+        "properties": {
+          "did": {
+            "type": "string",
+            "format": "did",
+            "description": "The DID of the repo."
+          }
+        }
+      },
+      "input": {
+        "encoding": "*/*"
+      }
+    }
+  }
+}
diff --git a/lexicons/com/atproto/temp/transferAccount.json b/lexicons/com/atproto/temp/transferAccount.json
@@ -0,0 +1,44 @@
+{
+  "lexicon": 1,
+  "id": "com.atproto.temp.transferAccount",
+  "defs": {
+    "main": {
+      "type": "procedure",
+      "description": "Transfer an account.",
+      "input": {
+        "encoding": "application/json",
+        "schema": {
+          "type": "object",
+          "required": ["handle", "did", "plcOp"],
+          "properties": {
+            "handle": { "type": "string", "format": "handle" },
+            "did": { "type": "string", "format": "did" },
+            "plcOp": { "type": "unknown" }
+          }
+        }
+      },
+      "output": {
+        "encoding": "application/json",
+        "schema": {
+          "type": "object",
+          "required": ["accessJwt", "refreshJwt", "handle", "did"],
+          "properties": {
+            "accessJwt": { "type": "string" },
+            "refreshJwt": { "type": "string" },
+            "handle": { "type": "string", "format": "handle" },
+            "did": { "type": "string", "format": "did" }
+          }
+        }
+      },
+      "errors": [
+        { "name": "InvalidHandle" },
+        { "name": "InvalidPassword" },
+        { "name": "InvalidInviteCode" },
+        { "name": "HandleNotAvailable" },
+        { "name": "UnsupportedDomain" },
+        { "name": "UnresolvableDid" },
+        { "name": "IncompatibleDidDoc" }
+      ]
+    }
+  }
+}
diff --git a/packages/api/src/client/index.ts b/packages/api/src/client/index.ts
@@ -8,6 +8,7 @@ import {
 import { schemas } from './lexicons'
 import { CID } from 'multiformats/cid'
 import * as ComAtprotoAdminDefs from './types/com/atproto/admin/defs'
+import * as ComAtprotoAdminDeleteAccount from './types/com/atproto/admin/deleteAccount'
 import * as ComAtprotoAdminDisableAccountInvites from './types/com/atproto/admin/disableAccountInvites'
 import * as ComAtprotoAdminDisableInviteCodes from './types/com/atproto/admin/disableInviteCodes'
 import * as ComAtprotoAdminEmitModerationEvent from './types/com/atproto/admin/emitModerationEvent'
@@ -76,6 +77,9 @@ import * as ComAtprotoSyncNotifyOfUpdate from './types/com/atproto/sync/notifyOf
 import * as ComAtprotoSyncRequestCrawl from './types/com/atproto/sync/requestCrawl'
 import * as ComAtprotoSyncSubscribeRepos from './types/com/atproto/sync/subscribeRepos'
 import * as ComAtprotoTempFetchLabels from './types/com/atproto/temp/fetchLabels'
+import * as ComAtprotoTempImportRepo from './types/com/atproto/temp/importRepo'
+import * as ComAtprotoTempPushBlob from './types/com/atproto/temp/pushBlob'
+import * as ComAtprotoTempTransferAccount from './types/com/atproto/temp/transferAccount'
 import * as AppBskyActorDefs from './types/app/bsky/actor/defs'
 import * as AppBskyActorGetPreferences from './types/app/bsky/actor/getPreferences'
 import * as AppBskyActorGetProfile from './types/app/bsky/actor/getProfile'
@@ -143,6 +147,7 @@ import * as AppBskyUnspeccedSearchActorsSkeleton from './types/app/bsky/unspecce
 import * as AppBskyUnspeccedSearchPostsSkeleton from './types/app/bsky/unspecced/searchPostsSkeleton'
 
 export * as ComAtprotoAdminDefs from './types/com/atproto/admin/defs'
+export * as ComAtprotoAdminDeleteAccount from './types/com/atproto/admin/deleteAccount'
 export * as ComAtprotoAdminDisableAccountInvites from './types/com/atproto/admin/disableAccountInvites'
 export * as ComAtprotoAdminDisableInviteCodes from './types/com/atproto/admin/disableInviteCodes'
 export * as ComAtprotoAdminEmitModerationEvent from './types/com/atproto/admin/emitModerationEvent'
@@ -211,6 +216,9 @@ export * as ComAtprotoSyncNotifyOfUpdate from './types/com/atproto/sync/notifyOf
 export * as ComAtprotoSyncRequestCrawl from './types/com/atproto/sync/requestCrawl'
 export * as ComAtprotoSyncSubscribeRepos from './types/com/atproto/sync/subscribeRepos'
 export * as ComAtprotoTempFetchLabels from './types/com/atproto/temp/fetchLabels'
+export * as ComAtprotoTempImportRepo from './types/com/atproto/temp/importRepo'
+export * as ComAtprotoTempPushBlob from './types/com/atproto/temp/pushBlob'
+export * as ComAtprotoTempTransferAccount from './types/com/atproto/temp/transferAccount'
 export * as AppBskyActorDefs from './types/app/bsky/actor/defs'
 export * as AppBskyActorGetPreferences from './types/app/bsky/actor/getPreferences'
 export * as AppBskyActorGetProfile from './types/app/bsky/actor/getProfile'
@@ -366,6 +374,17 @@ export class AdminNS {
     this._service = service
   }
 
+  deleteAccount(
+    data?: ComAtprotoAdminDeleteAccount.InputSchema,
+    opts?: ComAtprotoAdminDeleteAccount.CallOptions,
+  ): Promise<ComAtprotoAdminDeleteAccount.Response> {
+    return this._service.xrpc
+      .call('com.atproto.admin.deleteAccount', opts?.qp, data, opts)
+      .catch((e) => {
+        throw ComAtprotoAdminDeleteAccount.toKnownErr(e)
+      })
+  }
+
   disableAccountInvites(
     data?: ComAtprotoAdminDisableAccountInvites.InputSchema,
     opts?: ComAtprotoAdminDisableAccountInvites.CallOptions,
@@ -1108,6 +1127,39 @@ export class TempNS {
         throw ComAtprotoTempFetchLabels.toKnownErr(e)
       })
   }
+
+  importRepo(
+    data?: ComAtprotoTempImportRepo.InputSchema,
+    opts?: ComAtprotoTempImportRepo.CallOptions,
+  ): Promise<ComAtprotoTempImportRepo.Response> {
+    return this._service.xrpc
+      .call('com.atproto.temp.importRepo', opts?.qp, data, opts)
+      .catch((e) => {
+        throw ComAtprotoTempImportRepo.toKnownErr(e)
+      })
+  }
+
+  pushBlob(
+    data?: ComAtprotoTempPushBlob.InputSchema,
+    opts?: ComAtprotoTempPushBlob.CallOptions,
+  ): Promise<ComAtprotoTempPushBlob.Response> {
+    return this._service.xrpc
+      .call('com.atproto.temp.pushBlob', opts?.qp, data, opts)
+      .catch((e) => {
+        throw ComAtprotoTempPushBlob.toKnownErr(e)
+      })
+  }
+
+  transferAccount(
+    data?: ComAtprotoTempTransferAccount.InputSchema,
+    opts?: ComAtprotoTempTransferAccount.CallOptions,
+  ): Promise<ComAtprotoTempTransferAccount.Response> {
+    return this._service.xrpc
+      .call('com.atproto.temp.transferAccount', opts?.qp, data, opts)
+      .catch((e) => {
+        throw ComAtprotoTempTransferAccount.toKnownErr(e)
+      })
+  }
 }
 
 export class AppNS {

diff --git a/packages/api/src/client/lexicons.ts b/packages/api/src/client/lexicons.ts
@@ -820,6 +820,29 @@ export const schemaDict = {
       },
     },
   },
+  ComAtprotoAdminDeleteAccount: {
+    lexicon: 1,
+    id: 'com.atproto.admin.deleteAccount',
+    defs: {
+      main: {
+        type: 'procedure',
+        description: 'Delete a user account as an administrator.',
+        input: {
+          encoding: 'application/json',
+          schema: {
+            type: 'object',
+            required: ['did'],
+            properties: {
+              did: {
+                type: 'string',
+                format: 'did',
+              },
+            },
+          },
+        },
+      },
+    },
+  },
   ComAtprotoAdminDisableAccountInvites: {
     lexicon: 1,
     id: 'com.atproto.admin.disableAccountInvites',
@@ -3979,6 +4002,135 @@ export const schemaDict = {
       },
     },
   },
+  ComAtprotoTempImportRepo: {
+    lexicon: 1,
+    id: 'com.atproto.temp.importRepo',
+    defs: {
+      main: {
+        type: 'procedure',
+        description:
+          "Gets the did's repo, optionally catching up from a specific revision.",
+        parameters: {
+          type: 'params',
+          required: ['did'],
+          properties: {
+            did: {
+              type: 'string',
+              format: 'did',
+              description: 'The DID of the repo.',
+            },
+          },
+        },
+        input: {
+          encoding: 'application/vnd.ipld.car',
+        },
+        output: {
+          encoding: 'text/plain',
+        },
+      },
+    },
+  },
+  ComAtprotoTempPushBlob: {
+    lexicon: 1,
+    id: 'com.atproto.temp.pushBlob',
+    defs: {
+      main: {
+        type: 'procedure',
+        description:
+          "Gets the did's repo, optionally catching up from a specific revision.",
+        parameters: {
+          type: 'params',
+          required: ['did'],
+          properties: {
+            did: {
+              type: 'string',
+              format: 'did',
+              description: 'The DID of the repo.',
+            },
+          },
+        },
+        input: {
+          encoding: '*/*',
+        },
+      },
+    },
+  },
+  ComAtprotoTempTransferAccount: {
+    lexicon: 1,
+    id: 'com.atproto.temp.transferAccount',
+    defs: {
+      main: {
+        type: 'procedure',
+        description: 'Transfer an account.',
+        input: {
+          encoding: 'application/json',
+          schema: {
+            type: 'object',
+            required: ['handle', 'did', 'plcOp'],
+            properties: {
+              handle: {
+                type: 'string',
+                format: 'handle',
+              },
+              did: {
+                type: 'string',
+                format: 'did',
+              },
+              plcOp: {
+                type: 'unknown',
+              },
+            },
+          },
+        },
+        output: {
+          encoding: 'application/json',
+          schema: {
+            type: 'object',
+            required: ['accessJwt', 'refreshJwt', 'handle', 'did'],
+            properties: {
+              accessJwt: {
+                type: 'string',
+              },
+              refreshJwt: {
+                type: 'string',
+              },
+              handle: {
+                type: 'string',
+                format: 'handle',
+              },
+              did: {
+                type: 'string',
+                format: 'did',
+              },
+            },
+          },
+        },
+        errors: [
+          {
+            name: 'InvalidHandle',
+          },
+          {
+            name: 'InvalidPassword',
+          },
+          {
+            name: 'InvalidInviteCode',
+          },
+          {
+            name: 'HandleNotAvailable',
+          },
+          {
+            name: 'UnsupportedDomain',
+          },
+          {
+            name: 'UnresolvableDid',
+          },
+          {
+            name: 'IncompatibleDidDoc',
+          },
+        ],
+      },
+    },
+  },
   AppBskyActorDefs: {
     lexicon: 1,
     id: 'app.bsky.actor.defs',
@@ -7671,6 +7823,7 @@ export const schemas: LexiconDoc[] = Object.values(schemaDict) as LexiconDoc[]
 export const lexicons: Lexicons = new Lexicons(schemas)
 export const ids = {
   ComAtprotoAdminDefs: 'com.atproto.admin.defs',
+  ComAtprotoAdminDeleteAccount: 'com.atproto.admin.deleteAccount',
   ComAtprotoAdminDisableAccountInvites:
     'com.atproto.admin.disableAccountInvites',
   ComAtprotoAdminDisableInviteCodes: 'com.atproto.admin.disableInviteCodes',
@@ -7746,6 +7899,9 @@ export const ids = {
   ComAtprotoSyncRequestCrawl: 'com.atproto.sync.requestCrawl',
   ComAtprotoSyncSubscribeRepos: 'com.atproto.sync.subscribeRepos',
   ComAtprotoTempFetchLabels: 'com.atproto.temp.fetchLabels',
+  ComAtprotoTempImportRepo: 'com.atproto.temp.importRepo',
+  ComAtprotoTempPushBlob: 'com.atproto.temp.pushBlob',
+  ComAtprotoTempTransferAccount: 'com.atproto.temp.transferAccount',
   AppBskyActorDefs: 'app.bsky.actor.defs',
   AppBskyActorGetPreferences: 'app.bsky.actor.getPreferences',
   AppBskyActorGetProfile: 'app.bsky.actor.getProfile',