Hacker binary attack code.

High performance storage for your app: the Storage Foundation API

High performance storage for your app: the Storage Foundation API

The Storage Foundation API resembles a basic file system,
with direct access to stored data through buffers and offsets. It gives
developers flexibility by providing generic, simple, and performant
primitives on which they can build higher-level components.

Updated

The Storage Foundation API is part of the capabilities project and is currently in development. This post will be updated as the implementation progresses.

The web platform increasingly offers developers the tools they need to build fined-tuned high-performance applications for the web. Most notably, WebAssembly (Wasm) has opened the door to fast and powerful web applications, while technologies like Emscripten now allow developers to reuse tried and tested code on the web. To truly leverage this potential, developers must have the same power and flexibility when it comes to storage.

This is where the Storage Foundation API comes in. The Storage Foundation API is a new fast and unopinionated storage API that unlocks new and much-requested use cases for the web, such as implementing performant databases and gracefully managing large temporary files. With this new interface, developers can "bring their own storage" to the web, reducing the feature gap between web and platform-specific code.

The Storage Foundation API is designed to resemble a very basic file system so it gives developers flexibility by providing generic, simple, and performant primitives on which they can build higher-level components. Applications can take advantage of the best tool for their needs, finding the right balance between usability, performance, and reliability.

Why does the web need another storage API? #

The web platform offers a number of storage options for developers, each of which is built with specific use-cases in mind.

  • Some of these options clearly do not overlap with this proposal as they only allow very small amounts of data to be stored, like cookies, or the Web Storage API consisting of the sessionStorage and the localStorage mechanisms.
  • Other options are already deprecated for various reasons like the File and Directory Entries API or WebSQL.
  • The File System Access API has a similar API surface, but its use is to interface with the client's file system and provide access to data that may be outside of the origin's or even the browser's ownership. This different focus comes with stricter security considerations and higher performance costs.
  • The IndexedDB API can be used as a backend for some of the Storage Foundation API's use-cases. For example, Emscripten includes IDBFS, an IndexedDB-based persistent file system. However, since IndexedDB is fundamentally a key-value store, it comes with significant performance limitations. Furthermore, directly accessing subsections of a file is even more difficult and slower under IndexedDB.
  • Finally, the CacheStorage interface is widely supported and is tuned for storing large-sized data such as web application resources, but the values are immutable.

The Storage Foundation API is an attempt at closing all the gaps of the previous storage options by allowing for the performant storage of mutable large files defined within the origin of the application.

Suggested use cases for the Storage Foundation API #

Examples of sites that may use this API include:

  • Productivity or creativity apps that operate on large amounts of video, audio, or image data. Such apps can offload segments to disk instead of holding them in memory.
  • Apps that rely on a persistent file system accessible from Wasm and that need more performance than what IDBFS can guarantee.

What is the Storage Foundation API? #

There are two main parts to the API:

  • File system calls, which provide basic functionality to interact with files and file paths.
  • File handles, which provide read and write access to an existing file.

File system calls #

The Storage Foundation API introduces a new object, storageFoundation, that lives on the window object and that includes a number of functions:

We are currently exploring the tradeoffs between providing a synchronous versus an asynchronous API. The interfaces are designed to be asynchronous as a temporary measure and will be updated once a decision has been reached. For more background on the tradeoffs, see the Explainer.

  • storageFoundation.open(name): Opens the file with the given name if it exists and otherwise creates a new file. Returns a promise that resolves with the the opened file.

Warning: File names are restricted to lowercase alphanumeric characters and underscore (a-z, 0-9, _).

A file can only be opened once. This means concurrent access from different tabs is currently not possible.

  • storageFoundation.delete(name): Removes the file with the given name. Returns a promise that resolves when the file is deleted.
  • storageFoundation.rename(oldName, newName): Renames the file from the old name to the new name atomically. Returns a promise that resolves when the file is renamed.
  • storageFoundation.getAll(): Returns a promise that resolves with an array of all existing file names.
  • storageFoundation.requestCapacity(requestedCapacity): Requests new capacity (in bytes) for usage by the current execution context. Returns a promise that resolved with the remaining amount of capacity available.

The Storage Foundation API achieves fast and predictable performance by implementing its own quota management system. Web applications must explicitly ask for capacity before storing any new data. This request will be granted according to the browser's quota guidelines. Anytime an application starts a new JavaScript execution context (e.g., a new tab, a new worker, or when reloading the page), it must make sure it owns sufficient capacity before writing any data.

  • storageFoundation.releaseCapacity(toBeReleasedCapacity): Releases the specified number of bytes from the current execution context, and returns a promise that resolves with the remaining capacity.
  • storageFoundation.getRemainingCapacity(): Returns a promise that resolves with the capacity available for the current execution context.

File handles #

Working with files happens via the following functions:

The Storage Foundation API used to be called NativeIO. Some references to this name still remain and will be removed eventually.

  • NativeIOFile.close(): Closes a file, and returns a promise that resolves when the operation completes.
  • NativeIOFile.flush(): Synchronizes (that is, flushes) a file's in-memory state with the storage device, and returns a promise that resolves when the operation completes.

It is a known issue that flush() might be slow and we are exploring whether offering a faster, less reliable variant would be useful.

  • NativeIOFile.getLength(): Returns a promise that resolves with the length of the file in bytes.

  • NativeIOFile.setLength(length): Sets the length of the file in bytes, and returns a promise that resolves when the operation completes. If the new length is smaller than the current length, bytes are removed starting from the end of the file. Otherwise the file is extended with zero-valued bytes.

  • NativeIOFile.read(buffer, offset): Reads the contents of the file at the given offset through a buffer that is the result of transferring the given buffer, which is then left detached. Returns a NativeIOReadResult with the transferred buffer and the the number of bytes that were successfully read.

    A NativeIOReadResult is an object that consists of two entries:

    • buffer: An ArrayBufferView, which is the result of transferring the buffer passed to read(). It is of the same type and length as source buffer.
    • readBytes: The number of bytes that were successfully read into buffer. This may be less than the buffer size, if an error occurs or if the read range spans beyond the end of the file. It is set to zero if the read range is beyond the end of the file.
  • NativeIOFile.write(buffer, offset): Writes the contents of the given buffer into the file at the given offset. The buffer is transferred before any data is written and is therefore left detached. Returns a NativeIOWriteResult with the transferred buffer and the number of bytes that were successfully written. The file will be extended if the write range exceeds its length.

    A NativeIOWriteResult is an object that consists of two entries:

    • buffer: An ArrayBufferView which is the result of transferring the buffer passed to write(). It is of the same type and length as the source buffer.
    • writtenBytes: The number of bytes that were successfully written into buffer. This may be less than the buffer size if an error occurs.

Calls to NativeIOFile.write() only guarantee that the data has been written to the file, but it does not guarantee that the data has been persisted to the underlying storage. To ensure that no data loss occurs on system crash, you must call NativeIOFile.flush() and wait for it to successfully return.

Current status #

StepStatus
1. Create explainerComplete
2. Create initial draft of specificationNot started
3. Gather feedback & iterate on designIn progress
4. Origin trialIn progress
5. LaunchNot started

There is currently an ongoing effort to augment the origin private file system of the File System Access API as to not introduce yet another entry point for a storage system. This article will be updated as we make progress on this.

How to use the Storage Foundation API #

Enabling via about://flags #

To experiment with the Storage Foundation API locally, without an origin trial token, enable the #experimental-web-platform-features flag in about://flags.

Enabling support during the origin trial phase #

Starting in Chromium 90, the Storage Foundation API is available as an origin trial in Chromium. The origin trial is expected to end in Chromium 95 (November 10, 2021).

Origin trials allow you to try new features and give feedback on their usability, practicality, and effectiveness to the web standards community. For more information, see the Origin Trials Guide for Web Developers. To sign up for this or another origin trial, visit the registration page.

Register for the origin trial #

  1. Request a token for your origin.
  2. Add the token to your pages. There are two ways to do that:
    • Add an origin-trial <meta> tag to the head of each page. For example, this may look something like:
      <meta http-equiv="origin-trial" content="TOKEN_GOES_HERE">
    • If you can configure your server, you can also add the token using an Origin-Trial HTTP header. The resulting response header should look something like:
      Origin-Trial: TOKEN_GOES_HERE

Feature detection #

To check if the Storage Foundation API is supported, use:

if ('storageFoundation' in window) {
// The Storage Foundation API is supported.
}

Complete examples #

To make the concepts introduced above clearer, here are two complete examples that walk you through the different stages in the lifecycle of Storage Foundation files.

Opening, writing, reading, closing #

// Open a file (creating it if needed).
const file = await storageFoundation.open('test_file');
try {
// Request 100 bytes of capacity for this context.
await storageFoundation.requestCapacity(100);

const writeBuffer = new Uint8Array([64, 65, 66]);
// Write the buffer at offset 0. After this operation, `result.buffer`
// contains the transferred buffer and `result.writtenBytes` is 3,
// the number of bytes written. `writeBuffer` is left detached.
let result = await file.write(writeBuffer, 0);

const readBuffer = new Uint8Array(3);
// Read at offset 1. `result.buffer` contains the transferred buffer,
// `result.readBytes` is 2, the number of bytes read. `readBuffer` is left
// detached.
result = await file.read(readBuffer, 1);
// `Uint8Array(3) [65, 66, 0]`
console.log(result.buffer);
} finally {
file.close();
}

Opening, listing, deleting #

// Open three different files (creating them if needed).
await storageFoundation.open('sunrise');
await storageFoundation.open('noon');
await storageFoundation.open('sunset');
// List all existing files.
// `["sunset", "sunrise", "noon"]`
await storageFoundation.getAll();
// Delete one of the three files.
await storageFoundation.delete('noon');
// List all remaining existing files.
// `["sunrise", "noon"]`
await storageFoundation.getAll();

Demo #

You can play with the Storage Foundation API demo in the embed below. Create, rename, write into, and read from files, and see the available capacity you have requested update as you make changes. You can find the source code of the demo on Glitch.

Security and permissions #

The Chromium team has designed and implemented the Storage Foundation API using the core principles defined in Controlling Access to Powerful Web Platform Features, including user control, transparency, and ergonomics.

Following the same pattern as other modern storage APIs on the web, access to the Storage Foundation API is origin-bound, meaning that an origin may only access self-created data. It is also limited to secure contexts.

User control #

Storage quota will be used to distribute access to disk space and to prevent abuse. Memory you want to occupy needs to be requested first. Like other storage APIs, users can clear the space taken by Storage Foundation API through their browser.

Feedback #

The Chromium team wants to hear about your experiences with the Storage Foundation API.

Tell us about the API design #

Is there something about the API that does not work like you expected? Or are there missing methods or properties that you need to implement your idea? Have a question or comment on the security model? File a spec issue on the corresponding GitHub repo, or add your thoughts to an existing issue.

Report a problem with the implementation #

Did you find a bug with Chromium's implementation? Or is the implementation different from the spec? File a bug at new.crbug.com. Be sure to include as much detail as you can, simple instructions for reproducing, and enter Blink>Storage in the Components box. Glitch works great for sharing quick and easy repros.

Show support for the API #

Are you planning to use the Storage Foundation API? Your public support helps the Chromium team prioritize features and shows other browser vendors how critical it is to support them.

Send a tweet to @ChromiumDev using the hashtag #StorageFoundation and let us know where and how you are using it. Ask a question on StackOverflow with the hashtag #file-system-access-api.

Helpful links #

Acknowledgements #

The Storage Foundation API was specified and implemented by Emanuel Krivoy and Richard Stotz. This article was reviewed by Pete LePage and Joe Medley.

Hero image via Markus Spiske on Unsplash.

Last updated: Improve article