Login | Register   
RSS Feed
Download our iPhone app
Browse DevX
Sign up for e-mail newsletters from DevX

By submitting your information, you agree that devx.com may send you DevX offers via email, phone and text message, as well as email offers about other products and services that DevX believes may be of interest to you. DevX will process your information in accordance with the Quinstreet Privacy Policy.


Writing Data Safely with the CKPTFile Class : Page 2

Making changes to memory-mapped files is a delicate operation, but far from impossible. Learn how to make safe, atomic changes to memory-mapped files using a checkpoint system that will leave your applications fast and robust and your data impervious to corruption.




Application Security Testing: An Integral Part of DevOps

Atomic Writing
The goal is for the process of saving a file to be atomic. An atomic process is one that is either completely carried out or isn't carried out at all. In other words, partially finished jobs are not tolerated.

Furthermore, when a programmer defines an atomic process, it's imperative to include not only the fate of the new file, but the fate of the original file as well. The entire process needs to be atomic—either the entire new file is written to disk or the entire old file is left intact.

This is something that database engines do all the time. On a number of different levels, relational databases allow for certain activities to be done atomically. In particular, they allow transactions, which are clusters of data updates that are performed atomically. A transaction can be committed, which means that its updates are permanently stored; or a transaction can be rolled back, which means that none of its updates are stored.

Filesystems, on the other hand, don't do this as a matter of course. Files are written to the filesystem using commands such as open(), write(), and close() (or something similar). There's no command to roll back the changes made since the file was opened. With a few exceptions, filesystems aren't databases.

In practice, however, this isn't really a problem because of the way that programmers generally use the filesystem. For example, a program might write each new chunk of data to a different file. Or, you might take the trouble to write a document to a temporary file, and then replace the original file with the temporary one. Because replacing a file is generally atomic, this trick provides an easy way to get atomic writes using a filesystem and is commonly used by application developers.

Addressing Efficiency
Writing out an entire file often lacks efficiency. For a word processor it might be fine, because the files aren't too large and they aren't written to disk too often. But let's say you were implementing some kind of high-throughput store. If you're trying to make atomic changes to a 100MB file, and you are doing it 100 times per second, then you aren't going to want to write the entire file out each time you make an atomic change.

The most common approach in this scenario is to design the system to cache its data in memory, which is a lot faster than disk. However, data can't remain in memory forever—you have to write it out to disk at some point and when you do, it needs to be atomic. This is called checkpointing.

But in the case of the 100MB file, it's simply too large to write out in its entirety. You have to make changes directly to the file. And because of that our data has become delicate again.

We can't use any of the brute-force solutions described above because they are designed for low-throughput situations. On the other hand, we are reluctant to make changes directly to our file because we are scared of catastrophic failures.

There's a solution to this problem of course and the solution lies in making the changes themselves atomic. There's no reason to accept that atomic processes have to be slow. Clearly, this must be so, since much of the world's business runs on efficient database systems that deal safely with lots of data.

The problem is that these systems are necessarily enormous, expensive pieces of software, based on decades of experience and testing. They use special algorithms and protocols—generally based on the order in which one makes file modifications—to preserve the integrity of their data. But I don't want to spend forty years working on my I/O routines, and neither do you; we just want some safety.

In the next section, I'll explain a method that centers around the CKPTFile class. This method gives a pretty reasonable tradeoff: It provides very safe atomic data writing, it's fast, and it's not too complicated.

Comment and Contribute






(Maximum characters: 1200). You have 1200 characters left.



Thanks for your registration, follow us on our social networks to keep up-to-date