Marco.org

This is going to be a problem:

(Screenshot by someone on Twitter two weeks ago. I can’t find the tweet now — sorry.)

Every iOS app has its own “home” directory where it can store files. Every file and directory that an app puts there, except anything in a Caches or tmp directory, gets backed up when you sync your device to iTunes.

Prior to iOS 5, the system never deleted the contents of Caches and tmp, so they were safe places for apps to put data that should always be available but could be redownloaded if the user did a complete restore or otherwise lost all data, and therefore shouldn’t be taking up space in backups and slowing down syncs.

In iOS 5, since iCloud backups are now possible, Apple has started cracking down on apps that store too much in any backed-up directory, such as Documents. Many developers have recently received emails from Apple like this:

In recent testing it appears that [your app] stores a fair amount of data in its Documents folder.

Since iCloud backups are performed daily over Wi-Fi for each user’s iOS device, it’s important to ensure the best possible user experience by minimizing the amount of data being stored by your app.

In addition to purchased music, apps, books, Camera roll, and device settings, everything in your app’s home directory, including its Documents folder, is backed up to iCloud.

Data stored in the application bundle itself, the caches directory, and the temp directory is not backed up to iCloud. Your app should store data in these locations according to the iCloud Data Storage Guidelines on <http://developer.apple.com/icloud/documentation/data-storage/>.

Please review these guidelines, make any required changes to your app, and submit an update to the App Store.

And that documentation page makes it pretty clear:

1. Only documents and other data that is user-generated, or that cannot otherwise be recreated by your application, should be stored in the <Application_Home>/Documents directory and will be automatically backed up by iCloud.
2. Data that can be downloaded again or regenerated should be stored in the <Application_Home>/Library/Caches directory. Examples of files you should put in the Caches directory include database cache files and downloadable content, such as that used by magazine, newspaper, and map applications.

Sounds easy: just move anything that can be redownloaded to Caches.

Instapaper has stored its downloaded articles in Caches for years, since I didn’t want to slow down iTunes syncing for my customers or enlarge their backups unnecessarily, and full restores don’t happen often enough for it to be a problem for most people. This new policy now locks me into using Caches: I no longer have a choice.

But in iOS 5, there’s an important change: Caches and tmp — the only two directories that aren’t backed up — are “cleaned” out when the device is low on space.

A handful of developers reported this problem happening to them with Instapaper before iOS 5 was even released to the public — I’m dreading the influx of reports about this now that iOS 5 is available to everyone.

There’s no longer anywhere to store files that don’t need to be backed up (or can’t be, by the new policy) but shouldn’t be randomly deleted. This is problematic for lots of apps, including this quick list off the top of my head:

• Instapaper and anything that saves web articles for offline reading
• Ebook and comic-book apps (including iBooks, if the rules apply to it)
• Podcast clients (the rules don’t apply to synced podcasts from iTunes)
• Offline Wikipedia apps
• Offline mapping programs

The common theme is offline. It’s easy to assume that this isn’t a big problem — that surely, anything downloadable can be redownloaded at any time. But that’s not the case.

A common scenario: an Instapaper customer is stocking up an iPad for a long flight. She syncs a bunch of movies and podcasts, downloads some magazines, and buys a few new games, leaving very little free space. Right before boarding, she remembers to download the newest issue of The Economist. (I think highly of my customers.) This causes free space to fall below the threshold that triggers the cleaner, which — in the background, unbeknownst to her — deletes everything that was saved in Instapaper. Later in the flight, with no internet connectivity, she goes to launch Instapaper and finds it completely empty.

(Last week, almost this exact scenario happened to one of my customers.)

It creates a terrible experience for everyone:

• She has nothing to read. (She already finished The Economist.)
• My app appears to have failed and deleted her data, which makes it seem unreliable and decreases her opinion of it. If I’m lucky, she’ll email support and I’ll at least get a chance to explain myself. She’ll probably either be quietly dissatisfied or leave a 1-star review in the App Store telling everyone else that my app is unreliable and deletes data without warning, which will decrease my sales (and Apple’s 30% of those sales).
• She had a terrible experience on her iPad, which now seems less reliable as a whole, which reflects poorly on Apple and iOS.

It gets worse as you consider how often redownloading data isn’t a good option or isn’t even possible:

• Devices that are offline during large parts of the day, such as iPod Touches and Wi-Fi iPads
• 3G devices roaming internationally
• Any devices, even on Wi-Fi, connected to a network with expensive data transfer or with a low monthly transfer limit (a common scenario outside the U.S.)
• Devices in rural areas, where even the best home “broadband” connections available are very slow and can’t redownload hundreds of megs quickly
• Devices that are about to be carried out of Wi-Fi range when the owner realizes that his content has been deleted but he’s about to be late for work or the kids are getting rained on after soccer practice or the dog is about to explode and he needs to leave right now

But even with available, fast, unlimited internet connectivity, randomly deleting an app’s data is still a problem:

When customers save an article with Instapaper, get a book in iBooks, or download a podcast with Instacast, they expect it to be there next time they launch the app. Even though it’s technically redownloadable, customers see that as their data — they put it there, and it’s theirs to remove if and when they see fit.

When the cleaner wipes it out, it appears that the app has failed and deleted their data. And customers won’t know that it’s an iOS 5 behavior — they’ll understandably blame the app developers. Even though it’s not our fault, it’s certainly going to become our problem.

There needs to be a file storage location that behaves the way Caches did before iOS 5: it’s not backed up to iTunes or iCloud, it’s not synced, but it’s also never deleted unless the app is deleted.

UPDATE: In iOS 5.0.1, Apple has added a method for developers to mark files outside of Caches and tmp as “do not back up”. Presumably, once developers adopt this as necessary, this problem should be solved.