Skip to content

Commit

Permalink
Lexicon refactor (bluesky-social#658)
Browse files Browse the repository at this point in the history
* remove return in test

* couple of fixups in other pacakges

* Add dummy checks to declaration and follow app migrations, remove paranoid join

* update db nsid migration

* Ensure there are writes in follow app migration

* Add dumy check to votes-to-likes app migration, tidy

* Ensure there are writes in vote-to-like app migration

* update migration name

* pr feedback

* count utf8 & grapheme length

* add maxUtf8

* siwtch max semantics

* plural

* update post schema

* added bytes & cid refs

* add ipld<>json

* fixin up a could tings

* Add app.bsky.richtext.facet, replace post entities with facets

* plural actors

* wip

* Setup backlinks table on pds

* wip

* send & recieve cids/bytes with xrpc

* Track backlinks when indexing records on pds

* handle ipld vals in xrpc server

* added cids & bytes to codegen

* In createRecord, add deletions to avoid duplicate likes/follows/reposts

* Tests and fixes for prevention of dupe follows, likes, reposts

* Backlink migration tidy

* cleanup dag json parser

* Fix dupe backlink inserts

* Tidy

* blob refs + codegen

* Make profile displayName optional

* Test view and updateProfile for empty display name

* working into pds

* Make aggregate counts optional on post and profile views

* Make viewer state optional on post view for consistency

* Remove deprecated myState field on profile view

* Tidy repo method descriptions

* tests & types & fixes

* Implementation and tests for putRecord

* Remove updateProfile method

* Update repo service so that head can be taken for update externally

* Lex updates for compare-and-swap records/commits

* Add error to lex for bad repo compare-and-swaps

* Improve update-at-head thru repo service

* common package

* Implement and test compare-and-swaps on repo write methods

* Use lex discriminator for applyWrites

* Remove post entity/facet index

* Update lex descriptions to clarify repo write semantics

* Make deleteRecord idempotent w/ tests

* cleanup

* fix things up

* adding more formats

* tests

* updating schema

* Only generate tid rkeys on pds, support literal rkeys on client

* Add backlink indexes

* Update format of post embed views, fix external uri validation

* fixing up tests

* Include embeds on record embeds

* cleanup

* Notify users when they are quoted

* Remove determineRkey indirection

* fix api tests

* support concatenated cbor

* integrating to server

* re-enable tests

* fix up tests

* Thread compare-and-swaps down into repo service rather than use pinned storage

* Tidy

* Update packages/common/tests/ipld-multi.test.ts

Co-authored-by: devin ivy <[email protected]>

* Update packages/lexicon/src/validators/formats.ts

Co-authored-by: devin ivy <[email protected]>

* pr feedback

* pr feedback

* Add postgres-specific migration path for missing profile display names

* Tidy/clarify deep embeds

* Tidy

* rm unused escape

* decrease crud race count

* update subscribeRepos lexicon

* Fix applyWrite lexicon re: collection fields

* sign post event type

* update cids & bytes json encoding

* update lex blob & cid-link types

* updated codegen & pds

* number -> float

* missed a couple

* remove old image constraints

* pr feedback + descripts

* no hardcoded port numbers

* remove separate tooLarge evt

* fix dumb build error

* fixin gup lex + xrpc server

* better parsing of message types

* dont mutate body in subscription

* bugfix in subscription

* rm commented out code

* init feature branch

* undo

* Remove old lexicons

* Remove creator from profile view

* wip

* rework seqs

* fixed up tests

* bug fixing

* sequence handles & notify in dbTxn

* tidy

* update lex to include times

* test syncing handle changes

* one more fix

* handle too big evts

* dont thread sequencer through everything

* Split common into server vs web-friendly versions

* Make lexicon, identifier web-safe using common-web

* Switch api package to be a browser build, fix identifier package for browser bundling

* Fix pds and repo for lexicon package changes, tidy

* Make common-web a browser build, tidy

* fixing up deps

* fix up test

* turn off caching in actions

* Standardize repo write interfaces around repo input

* Update repo write endpoints for repo input field

* Remove scene follows during app migration

* API package updates (bluesky-social#712)

* Add bsky agent and various sugars to the api package

* Add richtext library to api package

* Update richtext to use facets and deprecate entities

* Update richtext to use utf8 indices

* Richtext converts deprecated entity indices from utf16 locations to utf8 locations

* Add note about encodings in the lexicon

* Add RichText facet detection

* Remove dead code

* Add deprecation notices to lexicons

* Usability improvements to RichText

* Update the api package readme

* Add RichText#detectFacetsWithoutResolution

* Add upsertProfile to bsky-agent

* Update packages/pds/src/api/com/atproto/repo/applyWrites.ts

Co-authored-by: devin ivy <[email protected]>

* pr feedback

* fix flaky timing streaming tests

* simplify emptyPromise

* fixed up open handles

* fix missed repo syntax

* fix error in test from fkey constraint

* fix another api agent bug

* Embed consistency, add complex record embed

* Tidy embed lex descriptions

* rename pg schemas

* use swc for jest

* fix up deps

* cleanup

* Update pds indexing, views, tests for complex record embeds

* fixing up profile view semantics

* wip

* update snaps

* Rename embed.complexRecord to embed.recordWithMedia

* Tidy aroud record w/ media embeds

* Add grapheme utilities to api RichText (bluesky-social#720)

Co-authored-by: dholms <[email protected]>

* Fix: app.bsky.feed.getPostThread#... to app.bsky.feed.defs#... (bluesky-social#726)

* Update bskyagent to use repo param

* Minor typing fix

* Add exports to api package: blobref & lex/json converters (bluesky-social#727)

* Add exports to api package: BlobRef & lex/json converters

* Add an example react-native fetch handler

* Switch all lingering references of recordRef to strongRef

* Update lexicon for richtext facets to have multiple features, byte slice rather than text slice

* Implement multi-feature richtext facets on pds

* Update api package to use updated richtext facets

* Minor fixes to admin repo/record views

* Fix app migration exports, remove old app migration

* Fix: sort richtext facets so they can render correctly

* Disable app migration dummy checks that don't work on live deploy

* Optimize lex de/serialization using simple checks

* Tidy comment typos

* App migration to cleanup notifications for likes, follows, old scene notifs

* Fix notification reason for change from vote to like

---------

Co-authored-by: Devin Ivy <[email protected]>
Co-authored-by: Paul Frazee <[email protected]>
  • Loading branch information
3 people authored Mar 31, 2023
1 parent 4cf9eed commit 7f008c0
Show file tree
Hide file tree
Showing 587 changed files with 20,980 additions and 15,723 deletions.
4 changes: 0 additions & 4 deletions .github/workflows/repo.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -29,10 +29,6 @@ jobs:
with:
node-version: 18
cache: "yarn"
- uses: actions/cache@v3
with:
path: /tmp/jest_rt
key: ${{ hashFiles('**/yarn.lock') }}-${{ matrix.shard }}
- run: yarn install --frozen-lockfile
- run: yarn test:withFlags --maxWorkers=2 --shard=${{ matrix.shard }} --passWithNoTests
verify:
Expand Down
71 changes: 71 additions & 0 deletions lexicons/app/bsky/actor/defs.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,71 @@
{
"lexicon": 1,
"id": "app.bsky.actor.defs",
"description": "A reference to an actor in the network.",
"defs": {
"profileViewBasic": {
"type": "object",
"required": ["did", "handle"],
"properties": {
"did": {"type": "string", "format": "did"},
"handle": {"type": "string", "format": "handle"},
"displayName": {
"type": "string",
"maxLength": 64
},
"avatar": { "type": "string" },
"viewer": {"type": "ref", "ref": "#viewerState"}
}
},
"profileView": {
"type": "object",
"required": ["did", "handle"],
"properties": {
"did": {"type": "string", "format":"did"},
"handle": {"type": "string", "format":"handle"},
"displayName": {
"type": "string",
"maxLength": 64
},
"description": {
"type": "string",
"maxLength": 256
},
"avatar": { "type": "string" },
"indexedAt": {"type": "string", "format": "datetime"},
"viewer": {"type": "ref", "ref": "#viewerState"}
}
},
"profileViewDetailed": {
"type": "object",
"required": ["did", "handle"],
"properties": {
"did": {"type": "string", "format": "did"},
"handle": {"type": "string", "format": "handle"},
"displayName": {
"type": "string",
"maxLength": 64
},
"description": {
"type": "string",
"maxLength": 256
},
"avatar": { "type": "string" },
"banner": { "type": "string" },
"followersCount": {"type": "integer"},
"followsCount": {"type": "integer"},
"postsCount": {"type": "integer"},
"indexedAt": {"type": "string", "format": "datetime"},
"viewer": {"type": "ref", "ref": "#viewerState"}
}
},
"viewerState": {
"type": "object",
"properties": {
"muted": {"type": "boolean"},
"following": {"type": "string", "format": "at-uri"},
"followedBy": {"type": "string", "format": "at-uri"}
}
}
}
}
4 changes: 2 additions & 2 deletions lexicons/app/bsky/actor/getProfile.json
Original file line number Diff line number Diff line change
Expand Up @@ -8,12 +8,12 @@
"type": "params",
"required": ["actor"],
"properties": {
"actor": {"type": "string"}
"actor": {"type": "string", "format": "at-identifier"}
}
},
"output": {
"encoding": "application/json",
"schema": {"type": "ref", "ref": "app.bsky.actor.profile#view"}
"schema": {"type": "ref", "ref": "app.bsky.actor.defs#profileViewDetailed"}
}
}
}
Expand Down
4 changes: 2 additions & 2 deletions lexicons/app/bsky/actor/getProfiles.json
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@
"properties": {
"actors": {
"type": "array",
"items": {"type": "string"},
"items": {"type": "string", "format": "at-identifier"},
"maxLength": 25
}
}
Expand All @@ -23,7 +23,7 @@
"properties": {
"profiles": {
"type": "array",
"items": {"type": "ref", "ref": "app.bsky.actor.profile#view"}
"items": {"type": "ref", "ref": "app.bsky.actor.defs#profileViewDetailed"}
}
}
}
Expand Down
2 changes: 1 addition & 1 deletion lexicons/app/bsky/actor/getSuggestions.json
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@
"cursor": {"type": "string"},
"actors": {
"type": "array",
"items": {"type": "ref", "ref": "app.bsky.actor.profile#viewBasic"}
"items": {"type": "ref", "ref": "app.bsky.actor.defs#profileView"}
}
}
}
Expand Down
71 changes: 2 additions & 69 deletions lexicons/app/bsky/actor/profile.json
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,6 @@
"key": "literal:self",
"record": {
"type": "object",
"required": ["displayName"],
"properties": {
"displayName": {
"type": "string",
Expand All @@ -18,83 +17,17 @@
"maxLength": 256
},
"avatar": {
"type": "image",
"type": "blob",
"accept": ["image/png", "image/jpeg"],
"maxWidth": 2000,
"maxHeight": 2000,
"maxSize": 1000000
},
"banner": {
"type": "image",
"type": "blob",
"accept": ["image/png", "image/jpeg"],
"maxWidth": 6000,
"maxHeight": 2000,
"maxSize": 1000000
}
}
}
},
"view": {
"type": "object",
"required": ["did", "declaration", "handle", "creator", "followersCount", "followsCount", "postsCount"],
"properties": {
"did": {"type": "string"},
"declaration": {"type": "ref", "ref": "app.bsky.system.declRef"},
"handle": {"type": "string"},
"displayName": {
"type": "string",
"maxLength": 64
},
"description": {
"type": "string",
"maxLength": 256
},
"avatar": { "type": "string" },
"banner": { "type": "string" },
"followersCount": {"type": "integer"},
"followsCount": {"type": "integer"},
"postsCount": {"type": "integer"},
"creator": {"type": "string"},
"indexedAt": {"type": "datetime"},
"viewer": {"type": "ref", "ref": "#viewerState"},
"myState": {"type": "ref", "ref": "#myState", "description": "Deprecated"}
}
},
"viewBasic": {
"type": "object",
"required": ["did", "declaration", "handle"],
"properties": {
"did": {"type": "string"},
"declaration": {"type": "ref", "ref": "app.bsky.system.declRef"},
"handle": {"type": "string"},
"displayName": {
"type": "string",
"maxLength": 64
},
"description": {
"type": "string",
"maxLength": 256
},
"avatar": { "type": "string" },
"indexedAt": {"type": "datetime"},
"viewer": {"type": "ref", "ref": "#viewerState"}
}
},
"viewerState": {
"type": "object",
"properties": {
"muted": {"type": "boolean"},
"following": {"type": "string"},
"followedBy": {"type": "string"}
}
},
"myState": {
"type": "object",
"description": "Deprecated in favor of #viewerState",
"properties": {
"follow": {"type": "string"},
"muted": {"type": "boolean"}
}
}
}
}
38 changes: 0 additions & 38 deletions lexicons/app/bsky/actor/ref.json

This file was deleted.

Original file line number Diff line number Diff line change
@@ -1,28 +1,28 @@
{
"lexicon": 1,
"id": "app.bsky.actor.search",
"id": "app.bsky.actor.searchActors",
"defs": {
"main": {
"type": "query",
"description": "Find users matching search criteria.",
"description": "Find actors matching search criteria.",
"parameters": {
"type": "params",
"properties": {
"term": {"type": "string"},
"limit": {"type": "integer", "minimum": 1, "maximum": 100, "default": 50},
"before": {"type": "string"}
"cursor": {"type": "string"}
}
},
"output": {
"encoding": "application/json",
"schema": {
"type": "object",
"required": ["users"],
"required": ["actors"],
"properties": {
"cursor": {"type": "string"},
"users": {
"actors": {
"type": "array",
"items": {"type": "ref", "ref": "app.bsky.actor.profile#viewBasic"}
"items": {"type": "ref", "ref": "app.bsky.actor.defs#profileView"}
}
}
}
Expand Down
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
{
"lexicon": 1,
"id": "app.bsky.actor.searchTypeahead",
"id": "app.bsky.actor.searchActorsTypeahead",
"defs": {
"main": {
"type": "query",
"description": "Find user suggestions for a search term.",
"description": "Find actor suggestions for a search term.",
"parameters": {
"type": "params",
"properties": {
Expand All @@ -16,11 +16,11 @@
"encoding": "application/json",
"schema": {
"type": "object",
"required": ["users"],
"required": ["actors"],
"properties": {
"users": {
"actors": {
"type": "array",
"items": {"type": "ref", "ref": "app.bsky.actor.ref#withInfo"}
"items": {"type": "ref", "ref": "app.bsky.actor.defs#profileViewBasic"}
}
}
}
Expand Down
59 changes: 0 additions & 59 deletions lexicons/app/bsky/actor/updateProfile.json

This file was deleted.

Loading

0 comments on commit 7f008c0

Please sign in to comment.