Nebula drives are real-time distributed storage for files and key value databases built on top of Hypercore Protocol. This project exists because the Telios email client needed a way to distribute and store encrypted emails on user's local file systems over a peer-to-peer (P2P) network. A lot of inspiration was taken from Hyperdrive, but Hyperdrive didn't have options for fine-grain access control, multiple writers, and the ability to delete files from disk once added to the drives.
Nebula drives come with a handful of useful features like:
- Shareable over company firewalls and mobile networks: The P2P network runs on Hyperswarm which has the ability to hole-punch through most company firewalls and mobile connections.
- Full Text Search: Create encrypted full text search indexes on top of Hypercores.
- Access Control: Control access to each file by sharing the file's hash and the drive's discovery key.
- Multiwriter: Drives can have multiple peers with write access by exchanging eachother's keys.
- Collections: Along with files, drives can create and share simple key value btree databases built on Hyperbee. Collections also have the option to be encrypted with a secret key.
- Connect to drives behind corporate firewalls and mobile networks
- Create and share key value databases between peers
- Upgrade multiwriter to Hypercore v10
- Build full text search indexes from encrypted Hypercores
- Upgrade collections API with Hyperbeedeebee
- Upgrade access control to limit sharing by a peer's public key
- Turn an existing directory into a drive and watch for changes
npm i @telios/nebula
/******************************************************
*
* Create a new drive and write encrypted files to it
*
******************************************************/
const Drive = require('@telios/nebula')
// Optionally pass in an encryption key to encrypt the drive's databases
const encryptionKey = Buffer.alloc(32, 'hello world')
const localDrive = new Drive(__dirname + "/drive", null, {
keyPair,
encryptionKey,
swarmOpts: {
server: true,
client: true
},
fullTextSearch: true
})
await localDrive.ready()
// Key to be shared with other devices or services that want to seed this drive
const drivePubKey = localDrive.publicKey
// Clone a remote drive
const remoteDrive = new Drive(__dirname + "/drive_remote", drivePubKey, {
keyPair,
swarmOpts: {
server: true,
client: true
}
})
await remoteDrive.ready()
localDrive.on('file-sync', file => {
// Local drive has synced somefile.json from remote drive
})
// Write a non-encrypted file to the drive
await remoteDrive.writeFile('/dest/path/on/drive/somefile.json', readableStream)
// Write anencrypted file to the drive
await remoteDrive.writeFile('/dest/path/on/drive/someEncryptedFile.json', readableStream, { encrypted: true })
/******************************************************************
*
* Create an encrypted and shared database with full text search
*
*****************************************************************/
const corpus = [
{
title: 'Painting 1',
text_body: "In your world you can create anything you desire."
},
{
title: 'Painting 2',
text_body: "I thought today we would make a happy little stream that's just running through the woods here."
},
{
title: 'Painting 3',
text_body: "See. We take the corner of the brush and let it play back-and-forth. No pressure. Just relax and watch it happen."
},
{
title: 'Painting 4',
text_body: "Just go back and put one little more happy tree in there. Without washing the brush, I'm gonna go right into some Van Dyke Brown."
},
{
title: 'Painting 5',
text_body: "Trees get lonely too, so we'll give him a little friend. If what you're doing doesn't make you happy - you're doing the wrong thing."
},
{
title: 'Painting 6',
text_body: "Son of a gun. We're not trying to teach you a thing to copy. We're just here to teach you a technique, then let you loose into the world."
}
]
const collection = await drive.db.collection('BobRoss')
for(const data of corpus) {
await collection.insert({ title: data.title, text_body: data.text_body })
}
// Build a search index from the title and text_body properties
await collection.ftsIndex(['title', 'text_body'])
// Query the index
const query = await collection.search("happy tree", { limit: 10 })
Create a drive to be shared over the network which can be replicated and seeded by other peers.
storagePath
: The directory where you want the drive to be created.key
: The public key of the remote drive you want to clone
Options include:
{
storage, // Override Hypercore's default random-access-file storage with a different random-access-storage module
storageMaxBytes, // Maximum bytes the drive will store before turning off replication/file syncing
encryptionKey, // optionally pass an encryption key to encrypt the drive's database
keyPair: { // ed25519 keypair
publicKey,
secretKey
},
joinSwarm: true | false // Optionally set whether or not to join hyperswarm when starting the drive. Defaults to true.
swarmOpts: { // Set server to true to start this drive as a server and announce its public key to the network
server: true | false,
client: true | false
},
checkNetworkStatus: true | false // Listen for when the drive's network status changes
fulltextSearch: true | false // support full text search indexes
blind: true // Initialize drive as a blind seeder. For example when you're seeding another encrypted drive and you don't have the encryption key
}
const Drive = require('nebula-drive')
// Create a new local drive.
const localDrive = new Drive(__dirname + "/drive", null, {
keyPair,
swarmOpts: {
server: true,
client: true
}
})
await localDrive.ready()
// Key to be shared with other devices or services that want to seed this drive
const drivePubKey = localDrive.publicKey
// Clone a remote drive
const remoteDrive = new Drive(__dirname + "/drive_remote", drivePubKey, {
keyPair,
swarmOpts: {
server: true,
client: true
}
})
await remoteDrive.ready()
Initialize the drive and all resources needed.
Adds a remote drive as a new writer. After a peer has been added, the drive will automatically try to reconnect to this peer after every restart.
Example Usage:
// Local drive on Device A
const drive1 = new Drive(__dirname + "/drive", null, {
keyPair,
swarmOpts: {
server: true,
client: true
}
})
// Local drive on Device B
const drive2 = new Drive(__dirname + "/drive", null, {
keyPair,
swarmOpts: {
server: true,
client: true
}
})
await drive2.addPeer(drive1.publicKey)
Stop replicating with another drive peer.
Write a file from a readable stream. When choosing to encrypt a file, the encryption key will be passed back in the response. Each file is encrypted with a unique key which should be stored separately.
path
: Full path where the file resides on the local drivedir/to/my/file.jpg
readableStream
: Any readableStreamfs.createReadableStream()
Options include:
// When encrypted is true a key and header value will be returned after the file has been written
{
encrypted: true
}
Creates a readable stream of data from the requested file path.
path
: Full path where the file resides on the local drivedir/to/my/file.jpg
Drives with many files may not want to announce every file by it's hash due to network bandwidth limits. In this case, a drive has the option of sharing it's discoveryKey
which peers can use to connect to the drive and then make a request file hash request.
discoveryKey
: Remote drive's discovery keydrive.discoveryKey
which is used by peers to request resources from the drive.fileHash
: Hash of the file being requested on the remote drive.opts
: If a key and header are passed in then the return stream will be the deciphered datakey
: Encryption key used for deciphering the encrypted stream. This key is returned from thedrive.writeFile
method.header
: Needed for validating the encrypted stream. This gets returned fromdrive.writeFile()
.
If drive.fetchFileByDriveHash
is returning encrypted data, then decryptFileStream
will transform that stream and return a new stream of deciphered data.
stream
: Readable stream of encrypted datakey
: Encryption key used for deciphering the encrypted stream. This key is returned from thedrive.writeFile
method.header
: Needed for validating the encrypted stream. This gets returned fromdrive.writeFile()
.
Fetching files as a batch automatically chunks parallel requests in a fixed batch size so a drive can request as many files as it needs without impacting performance.
files
: Array of file objects with the following structurediscovery_key
: Remote drive's discovery keydrive.discoveryKey
which is used by peers to request resources from the drive.hash
: Hash of the file being requested on the remote drive.key
: Encryption key used for deciphering the encrypted stream. This key is returned from thedrive.writeFile
method.header
: Needed for validating the encrypted stream. This gets returned fromdrive.writeFile()
.
cb
: Callback method that runs after every file stream has been initialized. Use this for handling what to do with the individual file streams. Note that this should return a promise.
Example Usage:
await drive.fetchFileBatch(files, (stream, file) => {
return new Promise((resolve, reject) => {
const writeStream = fs.createWriteStream(`./${file.path}`)
pump(stream, writeStream, (err) => {
resolve()
})
})
})
Fully close the drive and all of it's resources.
Returns the drive's storage info
{
file_bytes: 0,
core_bytes: 0,
total_bytes: 0
}
Emitted when the drive has recieved a message from a peer.
peerPubKey
: Public key of the peer that sent the messagesocket
: The socket returned on this event can be used as a duplex stream for bi-directional communication with the connecting peer.socket.write
socket.on('data, data => {})
Emitted when a collection has received an update from a remote peer
item
collection
: The collection that was updated_id
: The updated document's _idauthor
: Public key of the peer who commited the update
Emitted when a new file has been added to a local drive.
file
: A file objectpath
: drive path the file was saved tohash
: Hash of the file
enc
: Passes back properties needed to decrypt the filekey
: Key needed to decrypt the fileheader
: Needed for validating the encrypted stream
Emitted when the drive has synced any remote data.
Emitted when the drive has synced remote a remote file.
Emitted when a file has been deleted on the drive.
Emitted when there has been an error downloading from the remote drive
Emitted when either the internet connection or the drive's connection to Hyperswarm has changed. The drive option checkNetworkStatus
must be set to true in order for these events to be emitted.
Returns:
network
internet
: true|falsedrive
: true|false
Drive databases mimic the MongoDB API. Full API documentation can be found on Hyperbeedeebee
Creates a new collection. Collections are automatically encrypted when a drive is instantiated with encryptionKey
(drive.encryptionKey
)
Inserts a new document into the collection.
Example:
const doc = await collection.insert({ name: 'alice', age: 37 })
// doc._id gets set to an ObjectId if you don't specify it
Search through all documents by query
Example:
const docs = await collection.find({ name: 'alice' })
Creates an index for a set of fields. This will speed up queries and is required for sorting by fields
Example:
const docs = await collection.createIndex(['name','address'])
Create a full text search index from a collection's properties
const docs = []
// Documents can only be added to search indexes after they've been inserted into a collection.
const doc = await collection.insert(item)
docs.push(doc)
await collection.ftsIndex(['address', 'first_name', 'last_name'], docs)
Query a searchable index
Options include:
{
limit: 10
}