Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add multi-file write support to the js and python sdks #451

Open
wants to merge 35 commits into
base: main
Choose a base branch
from
Open
Changes from 1 commit
Commits
Show all changes
35 commits
Select commit Hold shift + click to select a range
b2e80e1
Added multi-file write support
0div Oct 4, 2024
9e6bc17
address PR comments
0div Oct 4, 2024
966f0c7
[WIP] Cleanup
ValentaTomas Oct 5, 2024
eda88d0
address PR comments
0div Oct 7, 2024
a2d4ad7
boyscouting: fix some docstrings
0div Oct 7, 2024
a379a58
Add multi-file write support for python-sdk sync
0div Oct 7, 2024
608ef45
Use `@overload`
0div Oct 8, 2024
68ac038
merge beta
0div Oct 8, 2024
f010211
adapt multi file write tests for nested dirs
0div Oct 8, 2024
fbc4af4
allow passing empty array of files in python-sdk
0div Oct 8, 2024
2afb6d1
allow passing empty array of files in js-sdk
0div Oct 8, 2024
68e5efa
address PR comments
0div Oct 9, 2024
365af43
add extra tests to sandbox_sync write
0div Oct 9, 2024
7767d8f
updated js-sdk tests to check empty path behavior
0div Oct 9, 2024
f5cd1c0
add multifile upload to sanbox_async
0div Oct 9, 2024
834f84c
merge beta
0div Oct 10, 2024
2956a6e
better error messages in python-sdk
0div Oct 10, 2024
c277f9f
better error messages in js-sdk
0div Oct 10, 2024
86262f1
docstring for dataclass
0div Oct 10, 2024
97d0cc1
merge main
0div Dec 12, 2024
cf262ae
fix errors.ts comment
0div Dec 12, 2024
4ca2414
fixed typing syntax and watch tests
0div Dec 12, 2024
93dc1f9
update docs
0div Dec 12, 2024
741b329
add minor changeset
0div Dec 12, 2024
bbeeb04
upadte upload docs and improve read-write-docs
0div Dec 12, 2024
0e2a684
fix indentation in upload docs
0div Dec 13, 2024
b867fe4
Update apps/web/src/app/(docs)/docs/filesystem/read-write/page.mdx
0div Dec 13, 2024
e730810
Update apps/web/src/app/(docs)/docs/filesystem/read-write/page.mdx
0div Dec 13, 2024
d3b99fe
Update apps/web/src/app/(docs)/docs/filesystem/upload/page.mdx
0div Dec 13, 2024
424baec
Update apps/web/src/app/(docs)/docs/filesystem/upload/page.mdx
0div Dec 13, 2024
e73ad23
Update apps/web/src/app/(docs)/docs/filesystem/upload/page.mdx
0div Dec 13, 2024
d0fc09a
Update apps/web/src/app/(docs)/docs/filesystem/upload/page.mdx
0div Dec 13, 2024
1708cc0
Update apps/web/src/app/(docs)/docs/filesystem/upload/page.mdx
0div Dec 13, 2024
502c414
remove WriteData type in js-sdk
0div Dec 13, 2024
724fbc6
remove WriteData type in python-sdk
0div Dec 13, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
[WIP] Cleanup
  • Loading branch information
ValentaTomas committed Oct 5, 2024
commit 966f0c73afb4c58d6666f35cdb008f410786f714
60 changes: 15 additions & 45 deletions packages/js-sdk/src/sandbox/filesystem/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -82,56 +82,30 @@ export class Filesystem {
}

async write(path: string, data: WriteData, opts?: FilesystemRequestOpts): Promise<EntryInfo>
async write(files: { data: WriteData; filename: string }[], opts?: FilesystemRequestOpts): Promise<EntryInfo[]>
async write(pathOrFiles: string | { data: WriteData; filename: string }[], dataOrOpts?: WriteData | FilesystemRequestOpts, opts?: FilesystemRequestOpts): Promise<EntryInfo | EntryInfo[]> {
let path: string
let data: WriteData | { data: WriteData; filename: string }[]

if (typeof pathOrFiles === 'string') {
path = pathOrFiles
data = dataOrOpts as WriteData
} else {
path = ''
data = pathOrFiles
opts = dataOrOpts as FilesystemRequestOpts
}

let blobs: Blob[] = []
async write(files: { path: string, data: WriteData }[], opts?: FilesystemRequestOpts): Promise<EntryInfo[]>
async write(pathOrFiles: string | { path: string; data: WriteData }[], dataOrOpts?: WriteData | FilesystemRequestOpts, opts?: FilesystemRequestOpts): Promise<EntryInfo | EntryInfo[]> {
const { path, writeOpts, writeFiles } = typeof pathOrFiles === 'string'
? { path: pathOrFiles, writeFiles: [{ data: dataOrOpts }], writeOpts: opts }
: { path: undefined, writeFiles: pathOrFiles, writeOpts: dataOrOpts }

0div marked this conversation as resolved.
Show resolved Hide resolved
if (Array.isArray(data)) {
for (const d of data) {
const blob = await new Response(d.data).blob()
blobs.push(blob)
}
} else {
const blob = await new Response(data).blob()
blobs.push(blob)
}
const blobs = await Promise.all(writeFiles.map(f => new Response(f.data).blob()))

const res = await this.envdApi.api.POST('/files', {
params: {
query: {
path,
username: opts?.user || defaultUsername,
username: writeOpts?.user || defaultUsername,
},
},
bodySerializer() {
const fd = new FormData()

for (let i = 0; i < blobs.length; i++) {
const blob = blobs[i]
if (Array.isArray(data)) {
const filename = data[i].filename
fd.append('file', blob, filename)
} else {
fd.append('file', blob)
}
}
return blobs.reduce((fd, blob, i) => {
fd.append('file', blob, writeFiles[i].path)

return fd
return fd
}, new FormData())
},
body: {},
signal: this.connectionConfig.getSignal(opts?.requestTimeoutMs),
signal: this.connectionConfig.getSignal(writeOpts?.requestTimeoutMs),
})

const err = await handleEnvdApiError(res)
Expand All @@ -144,11 +118,7 @@ export class Filesystem {
throw new Error('Expected to receive information about written file')
}

if (files.length > 1) {
return files as EntryInfo[]
} else {
return files[0] as EntryInfo
}
return files.length === 1 ? files[0] : files
}

async list(path: string, opts?: FilesystemRequestOpts): Promise<EntryInfo[]> {
Expand Down Expand Up @@ -274,8 +244,8 @@ export class Filesystem {

const reqTimeout = requestTimeoutMs
? setTimeout(() => {
controller.abort()
}, requestTimeoutMs)
controller.abort()
}, requestTimeoutMs)
: undefined

const events = this.rpc.watchDir(
Expand Down