The Costs of Living: How Market Freedom Erodes the Best Things in Life

I have to admit that I never finished reading this book. It was taken out of my hands by the great library strike of '07. I made it about half-way, but I found that the book had lost steam and was only throwing out an interesting anecdote every dozen pages by the middle. If you are curious about it, around 50% of the book's ideas are in the first chapter (or the second, I can't remember); the rest is just fodder for interest.

The thesis was that market freedoms erode utility.

In a free market, buyers vote with their dollars to signal to sellers what they want. If consumers want baggy pants, then sellers who make baggy pants get rewarded. If one seller makes the baggiest pants, then they will sell more, because consumers want the baggiest pants. If one seller can make baggy pants for less, they earn more profit either by selling more baggy pants because their's have a lower price, or by retaining more profit per (g-)unit.

This all works pretty well, but micro-economics is built on a couple of major assumptions:

• Rationality: a consumer chooses between alternatives to maximize utility assuming:
  • perfect knowledge: aware of all alternatives and their trade-offs in utility and price
  • competence: ability to evaluate alternatives
  • transitivity: if a=b and b=c then a=c
• Ordinality: choices can be ranked by level of utility

Swartz doesn't addresses these assumptions directly, but it's an underlying theme that some of them are false throughout his work.

Perfect knowledge is perhaps the simplest assumption to attack. Consumers simply do not have perfect knowledge, and produces regularly take advantage of this (otherwise their would be no need for advertising). The clearest example in the book was about restaurant kitchens. Imagine two restaurants where they appear identically equal in utility, except for the fact that one has a spotless kitchen, and the other has a dirty one. The cost of maintaining the clean kitchen puts the one restaurant at a disadvantage. Customers don't see the kitchen, so they don't know about it. Over time, the restaurant with the dirty kitchen will be more successful since it doesn't have the associated cleaning cost.

Bruce Schneier talked about lack of rationality (and asymmetrical information theory and The Market for Lemons) and how it relates to computer security in A Security Market for Lemons -- companies that spend R&D time on real security products get undercut by ones who don't because the consumers can't tell the difference.

The lack of rationality of consumers leads to a situation where economic profit becomes the only driver for producers. Making the best product doesn't matter if consumers don't have knowledge and if they can't competently evaluate alternatives. It becomes a game of who can convince consumer that they offer the best utility while cutting as many corners as possible. Ultimately, the market rewards whomever makes the most money, not who serves consumers best, so the producers race to the bottom.

The pursuit and exploitation of individual advantage in the service of profit is built into the ideology of the market. Those who fail to capitalize on their advantages will earn less money, or be fired by their bosses, or be driven out of business by their competitors.

You can see this in poisoned pet foods and leaded childrens' toys. Brands have become little more than advertising as everything is manufactured by outsourced low-cost producers.

The early part of the book focuses on attacking the application of market theory to social institutions. It argues that schooling, medicine and even baseball are debased when they are understood in terms of profit-making.

The medicine example is fairly long, but the gist of it was that doctors set up HMOs to serve their patients better. These got perverted by business types in the race to the bottom (the ones who didn't race went out of business because they weren't cost competitive with the ones that did), and they are left with the disaster of a health care system in the U.S. I don't remember the sport example.

Education is probably easier to explain. It talks about how the desire to quantify education has forced a market mentality into it. The need to quantify lead to metrics, and those metrics became twisted from being diagnostics to being the end goal. It goes like this: we need a standard test to asses education levels, then there is talk about evaluating teachers (and schools) based on how their students' score on the test (which was not its intent), so the teachers (and admin) respond by teaching to the test to maximize scores instead of working on general education. In the end, the standardized test score becomes the only thing that matters. This happened a lot at my old University in its quest to stay on top of the Macleans' University Rankings.

Another unfortunate example is how these metrics can extinguish the point of the activities. Students get rewarded based on number of books read, or something, so reading more (quantity) becomes the only goal. Students pick short books -- ones that are easy to read -- and read them as fast as possible. They don't think about what they read, they don't enjoy it, they don't reflect. They just read as fast as they can. The extrinsic reward, whatever it is, ultimately extinguishes the intrinsic motivation of the student.

Joel Spolsky talked about The Econ 101 Management Method to say the same things about software project management. Extrinsic motivation destroys intrinsic motivation, and what's worse is that your metric is never the thing that you really care about, but your employees end up only caring about the metric (since that's how they are measured).

Robert Austin, in his book Measuring and Managing Performance in Organizations, says there are two phases when you introduce new performance metrics. At first, you actually get what you wanted, because nobody has figured out how to cheat. In the second phase, you actually get something worse, as everyone figures out the trick to maximizing the thing that you’re measuring, even at the cost of ruining the company.

Unicode Mathematical Operator Entities

Math Symbols

Greek Letters

References

• Unicode Characters in the Mathematical Operators Block
• Unicode Mathematical Operators
• HTML Entities for symbols, mathematical symbols, and Greek letters
• Unicode character entities: Mathematical Operators
• Character entity references in HTML 4

Why Double-Checked Locking Doesn't Work

Consider a singleton:

public class Singleton {
    private static Singleton instance;
    private final Object important;
    protected Singleton() {
        important = Object.class;   // or something important
    }
    
    public static Singleton instance() {
        if(instance == null) {
            instance = new Singleton();
        }
        return instance;
    }
    
    public Object getImportant() { return important; }
}

This works well for a single threaded program, but there's a race in the instance() method. If one thread evaluates the if(instance == null) test and starts the Singleton() constructor, but is then pre-emepted, another thread can run the test again and evaluate it to false, and also run the constructor, breaking the singleton pattern.

A solution is to synchronize the instance() method.

    public static synchronized Singleton instance() {
        if(instance == null) {
            instance = new Singleton();
        }
        return instance;
    }

This eliminates the race (only one thread can be within a synchronized block), but making a synchronized call is expensive -- on the order of 100 times more than a regular call. The proposed solution to this problem is to use a double-checked lock:

    public static Singleton instance() {
        if(instance == null) {
            synchronized(this) {
                if(instance == null) {
                    instance = new Singleton();
                }
            }
        }
        return instance;
    }

Here, an inexpensive test is preformed before execution of the synchronized block. If the singleton has already been initialized, then the synchronized block does not execute, saving the synchronization overhead and maintaining the singleton pattern.

The problem with the double-checked lock is that it doesn't work without a strictly enforced memory model. Most systems do not have a strict enough memory model for it to work. A compiler (or VM, or whatever) can (and will) re-order instructions that it feels that it can reorder to improve performance, and the double-checked lock will fall victim to the reordering.

Without reordering, we expect execution to occur as it is written:

if(assigned to null) {
  synchronize
    if(assigned to null)
      initialize object
      assign object's memory location
  un-synchronize
return

But with reordering, the scheme can be broken:

if(assigned to null) {
  synchronize
    if(assigned to null)
      assign object's memory location
      initialize object
  un-synchronize
return

The difference is subtle. It's fine in a single-threaded context (which is why the compiler thinks that it can re-order), but it breaks the lock when multi-threaded. A thread can be pre-empted between the assignment and the initialization. Another thread can come and run the outer test, discover that the reference has been assigned a non-null value, and continue executing with a reference to an un-initialized object. This causes undefined (bad) behaviour in the second thread.

The best solution to this problem is to simply not use lazy initialization. Lazy initialization is sexy because it defers construction until it's needed. It saves memory (for a while) and delays execution (for a while), but the memory cost and execution time is paid eventually. If you statically initialize the singleton, it will all be paid for on the first reference to the class. This will make your start-up more expensive, but those costs (in space and time) would have been paid eventually.

A common objection to this is that with lazy initialization, you only pay for the init if the singleton is used. Static initialization still retains this property. The initialization will only occur if the class is loaded by the classloader (or linked by the linker, or loaded by the page-fault handler), and is therefore going to be in use. The prescribed (and preferred in most cases) solution is:

public class Singleton {
    private static Singleton instance = new Singleton();
    private final Object important;
    protected Singleton() {
        important = Object.class;   // or something important
    }
    
    public static Singleton instance() {
        return instance;
    }
    
    public Object getImportant() { return important; }
}

The second solution is to enforce the ordering of the instructions. As of Java 1.5, the volatile keyword prevents the reordering of reads/writes to a variable in relation to other variables (in addition to preventing local copies of variables). [The old behaviour was to enforce ordering on the variable itself, without respect to other variables, allowing the singleton to be assigned before its non-volatile instance variables were initialized.] Putting the volatile keyword on the singleton instance will prevent the variable from being assigned before its non-volatile instance variables are initialized:

    private volatile static Singleton instance;

    public static Singleton instance() {
        if(instance == null) {
            synchronized(this) {
                if(instance == null) {
                    instance = new Singleton();
                }
            }
        }
        return instance;
    }

Using the volatile keyword in itself is expensive. It forces the the thread to use the main-memory copy of any variables, preventing them from being stored in CPU registers, in non-coherent caches, and a host of other optimizations. If there is no thread contention for the resource, using volatile may cost as much as a lock (since there would be no signaling by the semaphore). In the volatile solution, this cost is paid for every method call, not just when locking is required. If there is low thread contention, it may cost as much as locking on every call.

Most developers are not familiar with the volatile keyword. It was mostly ignored pre-Java-1.5 by VM makers, and it didn't work as advertised. Even if it had worked, it's old semantics were not very useful. Accordingly, the development community largely ignored it. Relying on it for your singletons will risk naive developers from removing it, or following the double-checked lock pattern but missing the volatile keyword, or deploying the pattern to a pre-1.5 VM where it will not work. It should be avoided in favor of the statically initialized singleton.

[Aside: these arguments apply to C/C++ and other languages as well as Java.]

References

• Double-checked locking and the Singleton pattern
• Java theory and practice: Fixing the Java Memory Model, Part 2
• The "Double-Checked Locking is Broken" Declaration

SoftReference vs WeakReference

The difference between the a SoftReference and a WeakReference is how aggressively they are garbage collected. A WeakReference is GC'ed whenever the object becomes weakly reachable. A SoftReferenec is GC'ed when an object is softly reachable and the heap is full. Accordingly, WeakReferences are only useful as canonical mappings, and SoftReferences are most useful in memory-sensitive caches.

The WeakHashMap is used as a canonical map. Its keys are weakly held, not its values. There is no corresponding SoftHashMap to implement a cache included in the core API.