• Sign in

Marvel uses Dropbox webhooks for real-time syncing

Posted by Brendan Moore on December 09, 2014

Brendan Moore is the Co-Founder and CTO at Marvel, the simplest way to create cross-platform prototypes and bring digital app and web ideas to life. Founded in 2011 and based in London, Marvel is used by over 60,000 designers, product managers, teachers, and students.

One of the really powerful features in Marvel is the ability to provide real-time updates from our users' Dropbox accounts. Within the last month we upgraded from our well-serving, but now outdated, polling mechanism to real-time updates via Dropbox's new webhooks feature.

Polling for changes

Upon exploring the Dropbox Core API it became apparent that there was some pretty powerful features we were ignoring, including the delta feature. You can call delta with a string known as a cursor, essentially a state variable indicating where your system is in comparison to the user’s Dropbox. Dropbox then replies with a list of changes that have occurred since you were issued your cursor (e.g. files that have been altered, added or removed). Dropbox issues you a new cursor as well as a has_more indicator telling you if there are more changes to be synced. In essence this mechanism allows us to watch for file updates and changes for any content we have linked to a user’s Marvel account.

Excitedly Murat and I discussed how we could work this feature into Marvel and provide real-time syncing of files. Our goal was for a designer to be able to change a PSD in photoshop and a few moments later this change would reflect on their device. For the first 8 months of Marvel we used the following polling technique to achieve this:

  1. The user would load up their Marvel prototype on their device via an initial page load followed by an ajax call to retrieve project information. We’d send a saved copy of a cursor we received from Dropbox as the starting point of a user’s session. We then polled every 6 seconds to ask Dropbox for any changes within the user’s Dropbox.
  2. Each time we called Dropbox, it would respond with a potential list of file changes for the user. Any file changes not relating to the project we ignored. For any content related to the project we would pull and sync it to the Marvel project. The end result would be a new image synced to the user's prototype directly from Photoshop or their chosen design tool, such as Sketch.
  3. Finally, we’d save the most recent cursor, reissue it to the prototype, and reiterate the polling mechanism for as long as the user's session was opened.

The system served us well for months and we were reliably able to sync content within 20 seconds of saving a PSD file to a user's desktop. However, there were a number of drawbacks with this approach:

  1. Every time a project or prototype polled for changes we were opening a connection to our server and then to Dropbox, in a full synchronous request. This meant we were holding open connections for between 2-3 seconds. Given the frequency of the requests from prototypes this could at times potentially hold open way too many of our connections and prevent the website from operating normally.
  2. The sync was based on a poll-only mechanism which meant that we would only ever check for changes when the user came online or someone viewed their prototype.
  3. Any changes related to a file wouldn't update as quickly as desired, so precious seconds would be lost in between a potential update from Dropbox. In addition, our polling system sometimes polled multiple files in a prototype at a time, which proved to be inefficient.

Webhooks and Pusher

Webhooks are an efficient way for web platforms to notify each other of changes in real-time. Dropbox announced webhook support in May, immediately revolutionizing the way apps integrate with their service. Dropbox webhooks remove the need for Marvel to do expensive and inefficient polling for file changes. Instead, when a user alters a file in Dropbox our system gets notified that a change has occurred.

The call simply contains the user's Dropbox ID and our service queues up a check for file changes related to that user. Similar to step 2 in our polling solution, we can then utilise a cursor to verify any file changes and update that file within our system. In order to notify any prototypes or projects of this change we use Pusher, a London-based company that provides a fast and reliable broadcast method for messages to websites, apps, or any service you require. Pusher uses WebSockets to open a permanent connection to a Marvel prototype channel. Within seconds of receiving an update about a file change we can notify any relevant clients of that change.

Our aim is to continually improve the speed of this integration and to make the prototyping experience as real-time as possible.

The benefit

Dropbox's webhook and delta service has been invaluable for powering our service. Our users work in fast-paced client-driven environments where deliverables are always on tight timelines. An ability to leverage real-time syncing saves time and money when delivering ideas to a client. In it's simplest form, it's an empowering tool for designers to be able to validate their ideas without touching a single line of code.

It's a testament to the community of sharing and openness that's helped our startup to become a funded and fast growing business, powered by the Dropbox API.