Actually there are three allocation types:

• on demand - kind of "default" mode. You request an instance, if there is free capacity, you will get the instance. No long term commitment, but once you get an instance, it's yours. It may happen that you will get a message that there is no free capacity for the specific instance type and AZ (so far it happened to me only once with AWS).

• reserved - AWS reserves the capacity for you. You have guarantee that you will get the instance type in the selected region or AZ.

• spot instance - it's kind of auction / bidding of unused capacity. You ask for an instance, you provide your maximum price and if there is free capacity and your price is at the current price or higher, you will get an instance. The difference is - if the free capacity is exhausted, or the current price is higher than your maximum bid price, your spot instance is terminated . You can get a termination warning event upfront.

It is my understanding that Spot Instance pricing can be higher than On-Demand pricing in the case where a few users put in a high price to get access to a large number of servers for a short amount of time (usually only 1 hour). See page 18 of the Introduction to Spot Instances document:

Access Large Pools of Compute Capacity: Spot Instances can be used to help you meet occasional needs for large amounts of compute capacity (note that the default limit for Spot Instances is 100 versus the default limit of 20 for On-Demand Instances.) If your needs are urgent, you can specify a high maximum price (possibly even higher than the On-Demand price), which will raise your request’s relative priority and allow you to gain access to as much immediate capacity as possible given other requests and the Spot Instance capacity available at the time. While Spot Instances are generally not suitable for steadystate tasks such as serving web content, they can be used as a valuable source of instance capacity even for steady state applications when applications have urgent computing needs due to unanticipated or short-term demand spikes.

You can read more about spot instances and On-Demand vs Reserved vs Spot but basically here are your options:

1. If you need an always-on server for a long period of time (1 year or 3 years) choose a Reserved Instance for a "bulk discount". By paying up front Amazon can better estimate capacity needs and will guarantee that your server is available for the specified amount of time.

2. If you need a server that is always-on but for less than 1 year choose a On-Demand instance for your primary or master nodes. These take precedence over Spot instances and are almost always available whenever needed (up to 20, as indicated above). Once your instance is running it will not be terminated by Amazon unless the node has a hardware issue, in which case you will be moved to another node.

3. If your workload can benefit from extra instances but you do not want to pay the full On-Demand price then you can request Spot Instances for secondary or slave nodes. Spot instances are priced based on excess EC2 supply and spot instance demand, both of which are highly dynamic. There is no guarantee that a spot instance will run (even with a price higher than the On-Demand price), and your server instance can be terminated by Amazon at anytime (to be used by a higher paying Spot instance or an On-Demand instance). This requires your process to be re-entrant and that you save your work frequently.

There is a paper by folks from Israel Institute of Technology which analyzes the spot market. Their conclusion is that spot prices are not bid driven but rather generated with AR(1) function. So in such case it wouldn't be due to other's bids, but due to algorithm used.

http://www.cs.technion.ac.il/~ladypine/spotprice-ieee.pdf

Though, I guess it could have changed if spot market really reached the level where they could get nice prices from demand/supply. As Matt suggest it become a common practice to drive costs down with spot instances and many overbid on-demand prices to avoid down time. In longer run it was still cost effective ... but is it still? The real problem is that apart of Amazon nobody really knows how the prices are created.

This is actually a great example of people slightly abusing spot. People are saying 'Our workload is really important but we don't want to pay full on demand price', so they set a bid price higher than on-demand on the assumption that it is very unlikely to be terminated, but still want to get the 'cheapest possible' spot price on offer.

There have been cases where people enter, for example, $1000 (I've been told of at least one time this happened) because they want the benefit of the spot market. Of course naturally at some point the demand comes in and the spot price SOARS to make people pay higher than on-demand.

The way the spot market works is that Amazon have X instances spare capacity, and they count from the top down until they fill the need for all X instances. The 'price', then, is the lowest price at which they can fulfill those X instances.

So imagine Amazon have 10,000 instances - well they will count down to (say) $0.43 until they've got those 10,000 instances fulfilled. But if that supply suddenly drops to 100 instances, then maybe a few people put bid prices of $10,000 for their 100 instances, suddenly they'll be paying that $10k per hour.

Tl;dr Understand how spot works, and set a cap you are prepared to pay.