Goodnotes everywhere

For the last two years, the Goodnotes engineering team has been working on a project to bring the successful iPad notetaking app to other platforms. This case study covers how the 2022 iPad app of the year got to web, ChromeOS, Android, and Windows powered by web technologies and WebAssembly reusing the same Swift code the team has been working on for more than ten years.

Pedro Gómez

Thomas Steiner

Why Goodnotes came to web, Android, and Windows

In 2021 Goodnotes was only available as an app for iOS and iPad. The engineering team at Goodnotes accepted a huge technical challenge: creating a new version of Goodnotes but for additional operating systems and platforms. The product should be fully compatible with, and render the same notes as the iOS application. Any note taken on top of a PDF, or any image attached should be equivalent and show the same strokes the iOS app shows. Any stroke added should be equivalent to the one iOS users can create, independent from the tool the user was using—for example, pen, highlighter, fountain pen, shapes, or eraser.

Based on the requirements and the engineering team's experience, the team quickly concluded that reusing the Swift codebase would be the best course of action, given that it was already written and well-tested over many years. But why not just port the already existing iOS/iPad application to another platform or technology like Flutter or Compose Multiplatform? To move to a new platform would involve a rewrite of Goodnotes. Doing so might start a development race between the already-implemented iOS application and a to-be-built from zero new application, or involve stopping new development on the existing application while the new codebase caught up. If Goodnotes could reuse the Swift code, the team could benefit from new features implemented by the iOS team while the cross-platform team was working on the app fundamentals and reach feature parity.

The product had already solved a number of interesting challenges for iOS to add features like:

• Notes rendering.
• Documents and notes synchronization.
• Conflict resolution for notes using Conflict-Free Replicated Data Types.
• Data analysis for AI model evaluation.
• Content search and document indexing.
• Custom scrolling experience and animations.
• View model implementation for all the UI layers.

All of them would be far easier to implement for other platforms if the engineering team could get the iOS codebase already working for iOS and iPad applications and execute it as part of a project Goodnotes could ship as Windows, Android, or web applications.

Goodnotes' tech stack

Fortunately, there was a way to reuse the existing Swift code on the web—WebAssembly (Wasm). Goodnotes built a prototype using Wasm with the open source and community-maintained project SwiftWasm. With SwiftWasm the Goodnotes team could generate a Wasm binary using all the Swift code already implemented. This binary could be included in a web page shipped as a Progressive Web Application for Android, Windows, ChromeOS, and every other operating system.

The aim was to release Goodnotes as a PWA, and to be able to list it on every platform's store. In addition to Swift, the programming language already used for iOS, and WebAssembly used to execute Swift code on the web, the project used the following technologies:

• TypeScript: The most frequently used programming language for web technologies.
• React and webpack: The most popular framework and bundler for the web.
• PWA and service workers: Huge enablers for this project because the team could ship our app as an offline application that works like any other iOS app and you can install it from the store or the browser itself.
• PWABuilder: The main project Goodnotes use to wrap the PWA into a native Windows binary so the team can distribute our app from the Microsoft Store.
• Trusted Web Activities: The most important Android technology the company use to distribute our PWA as a native application under the hood.

The following figure shows what's implemented using classic TypeScript and React, and what's implemented using SwiftWasm and vanilla JavaScript, Swift, and WebAssembly. This part of the project uses JSKit, a JavaScript interoperability library for Swift and WebAssembly the team uses in order to handle the DOM in our editor screen from our Swift code when needed or even use some browser-specific APIs.

Why use Wasm and the web?

Even though Wasm is not officially supported by Apple, the following reasons are why the Goodnotes engineering team felt this approach was the best decision:

• The reuse of more than 100 thousand lines of code.
• The ability to continue development on the core product while also contributing to the cross-platform apps.
• The power of getting to every platform as soon as possible using an iterative development process.
• Having control to render the same document without duplicating all the business logic, and introducing differences in our implementations.
• Benefitting from all the performance improvements done on every platform at the same time (and all the bug fixes implemented on every platform).

The reuse of more than 100 thousand lines of code, and of the business logic implementing our rendering pipeline was fundamental. At the same time, making the Swift code compatible with other toolchains lets them reuse this code in different platforms in the future if needed.

Iterative product development

The team took an iterative approach in order to get something to users as quickly as possible. Goodnotes began with a read-only version of the product where users could get any shared document and read it from any platform. Just with a link, they would be able to access and read the same notes they wrote from their iPad. The next phase added in editing features, to make the cross-platform versions equivalent to the iOS one.

The first version of the read-only product took six months to develop, the following nine months were dedicated to the first bunch of editing features and the UI screen where you can check all the documents you created or somebody shared with you. In addition, new features of the iOS platform were easy to port to the cross-platform project thanks to the SwiftWasm Toolchain. As an example, a new type of pen was created and easily implemented cross-platform by reusing thousands of lines of code.

Building this project was an incredible experience, and Goodnotes has learned a lot from it. That's why the following sections will focus on interesting technical points about web development and the usage of WebAssembly and languages like Swift.

Initial obstacles

Working on this project was super challenging from many different points of view. The first obstacle the team found was related to the SwiftWasm toolchain. The toolchain was a huge enabler for the team, but not all iOS code was compatible with Wasm. For example, code related to IO or UI—like the implementation of views, API clients, or access to the database was not reusable, so the team needed to start refactoring specific parts of the app to be able to reuse them from the cross-platform solution. Most of the PRs the team created were refactors to abstract dependencies so the team could later replace them using dependency injection or other similar strategies. The iOS code originally mixed raw business logic that could be implemented in Wasm with code responsible for input/output and user interface that couldn't be implemented in Wasm because Wasm doesn't support either. So IO and UI code needed to be reimplemented in TypeScript once the Swift business logic was ready to be reused between platforms.

Performance problems solved

Once Goodnotes started working on the editor, the team identified some problems with the editing experience, and challenging technology constraints got into our roadmap. The first problem was related to performance. JavaScript is a single-threaded language. This means it has one call stack and one memory heap. It executes code in order and must finish executing a piece of code before moving on to the next. It's synchronous, but at times that can be harmful. For example, if a function takes a while to execute or has to wait on something, it freezes everything up in the meantime. And this is exactly what the engineers had to solve. Evaluating some specific paths in our codebase related to the rendering layer or other complex algorithms was a problem for the team, because these algorithms were synchronous, and executing them was blocking the main thread. The Goodnotes team rewrote them to make them faster, and refactored some of them to make them asynchronous. They also introduced a yield strategy so the app could stop the algorithm execution and continue it later, letting the browser update the UI and avoid dropping frames. This was not a problem for the iOS application because it can use threads and evaluate these algorithms in the background while the main iOS thread updates the user interface.

Another solution the engineering team had to solve was migrating a UI based on HTML elements attached to the DOM, to a document UI based on a full-screen canvas. The project started showing all the notes and content related to a document as part of the DOM structure using HTML elements as any other web page would do, but at some point migrated to a full-screen canvas to improve performance on low-end devices by reducing the time the browser is working on DOM updates.

The following changes were identified by the engineering team as things that could have reduced some of the issues encountered, had they done these at the beginning of the project.

• Offload the main thread more by using web workers frequently for heavy algorithms.
• Make usage of exported and imported functions instead of the JS-Swift interop library since the beginning so they can reduce the performance impact of getting out of the Wasm context. This JavaScript interop library is helpful to get access to the DOM or the browser but it is slower than native Wasm exported functions.
• Ensure the code allows the usage of OffscreenCanvas under the hood so the app can offload the main thread and move all usage of the Canvas API to a web worker maximizing applications' performance when writing notes.
• Move all the Wasm-related execution to a web worker or even a pool of web workers so the app can reduce the main thread workload.

The text editor

Another interesting problem was related to one specific tool, the text editor. The iOS implementation for this tool is based on NSAttributedString, a small toolset using RTF under the hood. However, this implementation is not compatible with SwiftWasm so the cross-platform team was forced to first create a custom parser based on the RTF grammar and later implement the editing experience by transforming RTF into HTML and vice versa. Meanwhile, the iOS team started working on the new implementation for this tool replacing the usage of RTF with a custom model so the app can represent styled text in a friendly way for all the platforms sharing the same Swift code.

This challenge was one of the most interesting points in the project roadmap because it was solved iteratively based on the user's needs. It was an engineering problem solved using a user-focused approach where the team needed to rewrite part of the code to be able to render text so they enabled text editing in a second release.

Iterative releases

The evolution of the project over the last two years has been incredible. The team started working on a read-only version of the project and months later shipped a brand new version with a lot of editing capabilities. To frequently release code changes to production the team decided to extensively use feature flags. For every release, the team could enable new features and also release code changes implementing new features the user would see weeks later. However, there is something the team thinks they could have improved! They think introducing a dynamic feature flag system would have helped speed things up, as it would remove the need for a redeploy to change flag values. This would give Goodnotes more flexibility and also speed up the deployment of the new feature because Goodnotes wouldn't need to link the project deployment to the product release.

Offline work

One of the major features the team worked on is offline support. Being able to edit your documents and modify them is one feature you would expect from any application like this. However, this is not a simple feature because Goodnotes supports collaboration. This means all the changes done by different users on different devices should end up on every device without asking users to solve any conflicts. Goodnotes solved this problem long ago by using CRDTs under the hood. Thanks to these Conflict-free Replicated Data Types, Goodnotes is able to combine all the changes done on any document by any user and merge the changes without any merge conflict. The usage of IndexedDB and the storage available for web browsers was a huge enabler for the collaborative offline experience on the web.

On top of that, opening the Goodnotes web app results in an initial upfront download cost of around 40MB because of the Wasm binary size. Initially, the Goodnotes team relied purely on the regular browser cache for the app bundle itself and most of the API endpoints they use, but in hindsight could have profited from the more reliable Cache API and service workers earlier. The team originally shied away from this task due to its assumed complexity, but in the end, realized that Workbox made it a lot less frightening.

Recommendations when using Swift on the web

If you have an iOS application with a lot of code you want to reuse, get ready because you are about to start an incredible journey. There are some tips you may find interesting before you start.

• Check what code you want to reuse. If your app's business logic is implemented on the server side, it is likely you'd love to reuse your UI code, and Wasm will not help you here. The team briefly looked at Tokamak, a SwiftUI-compatible framework for building browser apps with WebAssembly, but it wasn't mature enough for the app needs. However, if your app has strong business logic or algorithms implemented as part of the client code, Wasm will be your best friend.
• Ensure your Swift codebase is ready. Software design patterns for the UI layer or specific architectures creating a strong separation between your UI logic and your business logic will be really handy because you won't be able to reuse the UI layer implementation. Clean architecture or hexagonal architecture principles will be fundamental as well, because you'll have to inject and provide dependencies for all the IO-related code and it will be way easier to do if you follow these architectures where implementation details are defined as abstractions and the dependency inversion principle is heavily used.
• Wasm doesn't provide UI code. Therefore, decide on the UI framework you want to use for the web.
• JSKit will help you integrate your Swift code with JavaScript but keep in mind if you have a hotpath, crossing the JS–Swift bridge may be expensive and you would need to replace it with exported functions. You can learn more about how JSKit works under the hood in the official documentation and the Dynamic Member Lookup in Swift, a hidden gem! post.
• Whether you can reuse your architecture will depend on the architecture your app follows and the async code execution mechanism library you use. Patterns like MVVP or composable architecture will help you to reuse your view models and part of the UI logic without coupling the implementation to UIKit dependencies you can't use with Wasm. RXSwift and other libraries may not be compatible with Wasm, so keep it in mind because you'll have to use OpenCombine, async/await, and streams in Goodnotes' Swift code.
• Compress the Wasm binary using gzip or brotli. Keep in mind the size of the binary will be quite big for classic web applications.
• Even when you can use Wasm without the PWA, ensure you at least include a service worker, even if your web app has no manifest or you don't want the user to install it. The service worker will save and serve the Wasm binary for free and all the app resources so the user doesn't need to download them every time they open your project.
• Keep in mind hiring may be harder than expected. You may need to hire strong web developers with some experience on Swift or strong Swift developers with some experience on web. If you can find generalist engineers with some knowledge on both platforms, that would be awesome

Conclusions

Building a web project using a complex tech stack while working on a product full of challenges is an incredible experience. It's going to be hard, but totally worth it. Goodnotes could never have released a version for Windows, Android, ChromeOS, and web while working on new features for the iOS application without using this approach. Thanks to this tech stack and Goodnotes' engineering team, Goodnotes is now everywhere, and the team is ready to continue working on the next challenges! If you want to know more about this project, you can watch a talk the Goodnotes team gave at NSSpain 2023. Be sure to give Goodnotes for web a try!