There are other hardware costs.
Power, UPS, PDM's , Power Whips, AC, Floor space, Server cabinets, SNX, people, etc...
Most cloud based providers charge more when you scale so after a time it makes sense to do it your self and get others to piggy back on your infrastructure so you can charge them. Last I read 1.5% of the worlds power usage is social media data center infrastructure. That is one of the reasons why it made sense for Google to build a 3GW solar farm on their campus. This is also one of the reasons they really want you to use their cloud. You might get some usage free but they get paid through advertising or some other channel.
Not really a new paradigm. Cloud vs local has been going on for quite some time. Initially it was all cloud since only large institutions could afford a system. Then small systems came out and small data centers started to crop up. It has been going back and forth every since. What makes the cloud possible is the same thing that makes a very small data center possible.
The fight to a certain extent is talent. If you have a good IT person they can build a small data center in a small foot print. The tools keep making it all easier but since it is easier we get more features so it get more complex. Good Admins are not real easy to find. There is a ton of talent out there. Getting a person who is talented, organized, works hard, cares, communicates, proactive, documents, adaptive, level headed, and can prioritize is not so easy.