Marvel uses Dropbox webhooks for real-time syncing
Posted by Brendan Moore on December 09, 2014
Brendan Moore is the Co-Founder and CTO at Marvel, the simplest way to create cross-platform prototypes and bring digital app and web ideas to life. Founded in 2011 and based in London, Marvel is used by over 60,000 designers, product managers, teachers, and students.
One of the really powerful features in Marvel is the ability to provide real-time updates from our users' Dropbox accounts. Within the last month we upgraded from our well-serving, but now outdated, polling mechanism to real-time updates via Dropbox's new webhooks feature.
Polling for changes
Upon exploring the Dropbox Core API it became apparent that there was some pretty powerful features we were ignoring, including the delta feature. You can call delta with a string known as a cursor, essentially a state variable indicating where your system is in comparison to the user’s Dropbox. Dropbox then replies with a list of changes that have occurred since you were issued your cursor (e.g. files that have been altered, added or removed). Dropbox issues you a new cursor as well as a has_more indicator telling you if there are more changes to be synced. In essence this mechanism allows us to watch for file updates and changes for any content we have linked to a user’s Marvel account.
Excitedly Murat and I discussed how we could work this feature into Marvel and provide real-time syncing of files. Our goal was for a designer to be able to change a PSD in photoshop and a few moments later this change would reflect on their device. For the first 8 months of Marvel we used the following polling technique to achieve this:
- The user would load up their Marvel prototype on their device via an initial page load followed by an ajax call to retrieve project information. We’d send a saved copy of a cursor we received from Dropbox as the starting point of a user’s session. We then polled every 6 seconds to ask Dropbox for any changes within the user’s Dropbox.
- Each time we called Dropbox, it would respond with a potential list of file changes for the user. Any file changes not relating to the project we ignored. For any content related to the project we would pull and sync it to the Marvel project. The end result would be a new image synced to the user's prototype directly from Photoshop or their chosen design tool, such as Sketch.
- Finally, we’d save the most recent cursor, reissue it to the prototype, and reiterate the polling mechanism for as long as the user's session was opened.
The system served us well for months and we were reliably able to sync content within 20 seconds of saving a PSD file to a user's desktop. However, there were a number of drawbacks with this approach:
- Every time a project or prototype polled for changes we were opening a connection to our server and then to Dropbox, in a full synchronous request. This meant we were holding open connections for between 2-3 seconds. Given the frequency of the requests from prototypes this could at times potentially hold open way too many of our connections and prevent the website from operating normally.
- The sync was based on a poll-only mechanism which meant that we would only ever check for changes when the user came online or someone viewed their prototype.
- Any changes related to a file wouldn't update as quickly as desired, so precious seconds would be lost in between a potential update from Dropbox. In addition, our polling system sometimes polled multiple files in a prototype at a time, which proved to be inefficient.
Webhooks and Pusher
Webhooks are an efficient way for web platforms to notify each other of changes in real-time. Dropbox announced webhook support in May, immediately revolutionizing the way apps integrate with their service. Dropbox webhooks remove the need for Marvel to do expensive and inefficient polling for file changes. Instead, when a user alters a file in Dropbox our system gets notified that a change has occurred.
The call simply contains the user's Dropbox ID and our service queues up a check for file changes related to that user. Similar to step 2 in our polling solution, we can then utilise a cursor to verify any file changes and update that file within our system. In order to notify any prototypes or projects of this change we use Pusher, a London-based company that provides a fast and reliable broadcast method for messages to websites, apps, or any service you require. Pusher uses WebSockets to open a permanent connection to a Marvel prototype channel. Within seconds of receiving an update about a file change we can notify any relevant clients of that change.
Our aim is to continually improve the speed of this integration and to make the prototyping experience as real-time as possible.
The benefit
Dropbox's webhook and delta service has been invaluable for powering our service. Our users work in fast-paced client-driven environments where deliverables are always on tight timelines. An ability to leverage real-time syncing saves time and money when delivering ideas to a client. In it's simplest form, it's an empowering tool for designers to be able to validate their ideas without touching a single line of code.
It's a testament to the community of sharing and openness that's helped our startup to become a funded and fast growing business, powered by the Dropbox API.
Announcing the Dropbox for Business API
Posted by Leah Culver on December 03, 2014
Today we're excited to launch the new Dropbox for Business API.
Dropbox for Business provides additional storage and security features specifically for companies, and now these features are available via the API! If you're new to Dropbox for Business, you can find out more here.
New API endpoints
First, we've added lots of new API endpoints, specifically for businesses. Your app can now get info about your Dropbox for Business team and team members. Your app can also manage team members with endpoints to add, remove, and update your Dropbox for Business team. Want to know more about your team's activity? Check out the new reports and audit log API endpoints.
Member file access
In addition to new API endpoints, apps with Team member file access permission can make Core API calls on behalf of any member of a Dropbox for Business team. This allows your app to provide admins with additional security and monitoring features.
Webhooks
Two new types of webhook notifications are available for Dropbox for Business apps. First, your app can receive per-user notifications of updates for all members of a Dropbox for Business team, similar to the existing webhook notifications. Second, your app can receive notifications of changes to team membership such as when a user is invited to a team, a member's profile or permissions are updated, or a member is removed from a team. Find out more about Dropbox for Business webhooks.
Creating a Dropbox for Business app
To use the new Dropbox for Business API, you'll need to create a new app in the Dropbox App Console. You'll see that there are new permissions that allow access to certain Dropbox for Business API features.
Try out the new Dropbox for Business API and let us know what you think. We're excited to see what you build!
Heroku announces Dropbox Sync
Posted by Steve Marx on November 19, 2014
Today, Heroku announced the ability to deploy apps directly from Dropbox!
They're calling this feature Dropbox Sync, and it's available in beta right now. You can read the full details about the feature on the Heroku blog.
How it works
Heroku uses the /delta endpoint to determine what files have changed in Dropbox. Developers may then click a "Deploy" button in the Heroku dashboard to kick off a deployment.
Changes made by one user can be synced with a collaborator's Dropbox. Similarly, changes made via git (Heroku's other deployment protocol) are pushed to Dropbox and vice versa. This syncing back and forth could potentially lead to conflicts so Heroku is using Dropbox's built-in mechanism for flagging "conflicted copy" files, and then asking users to resolve such conflicts before deploying from Dropbox again. See the Dropbox Sync article in Heroku's Dev Center for more details on collaboration and conflict resolution.
You can do this too
Heroku built this feature themselves using the existing functionality in the Core API. We've seen quite a few companies build similar integrations with Dropbox to deploy code, content, or other assets. This kind of content syncing and deployment is a growing use case for the Dropbox API, so we're always working hard to make it even easier to build. For example, we recently added webhooks so each file save can trigger an event such as a code deploy.
Give it a try in your own product, and let us know if you run into trouble!
Sync and Datastore SDK 3.1.1
Posted by Leah Culver on October 10, 2014
We've released a new version of the Sync and Datastore SDK, version 3.1.1. This version includes no new features, but contains several bug fixes.
Specifically for iOS and OS X, the new SDK was built with the latest version of Xcode 6 and now properly raises a DBException instead of the generic NSException. See the full changelog for more details.
Download a new version of the Sync and Datastore SDK 3.1.1 for iOS, OS X, or Android today!
Lists app in Swift and Swift Hack Day
Posted by Leah Culver on October 08, 2014
A couple weekends ago, Dropbox participated in Swift Hack Day in San Francisco.
I got a chance to play around with the new Swift programming language for iOS and OS X and even created my own hack, a Swift version of our Lists sample app.
You can find the code for Swift Lists on GitHub.
I'm still new to Swift so check it out and let us know what you think on our developer forum!