In Code, Charles Petzold avoids the over simplifications of other books altogether. Petzold starts by explaining the use of Morse code to talk to friends using flashlights, and moves on to relays, logic gates, flip flops. With each chapter Petzold adds another layer to your knowledge. At the end of the book you will understand the inner workings of a computer from the simplest to most complex components. After reading this book, I have all the knowledge I'd need to build a simple computer from the ground up.
One of my favorite characteristics of Petzold's writing is his foolproof explanation and careful attention to detail. As examples, Petzold uses accurate and interesting real world scenarios. A particular favorite of mine was his description of how a barcode works, to help explain the binary system. Because Code explains things from the very lowest levels first and one step at a time, very little previous knowledge of computers is required. I've enjoyed reading this book so much - hopefully you will to! I highly recommend it to anyone wanting to know what really goes on inside the computer.
I might even start and post a project where I actually build a computer from scratch. For the time being, here is a very interesting link to a webpage describing such a project: http://www.homebrewcpu.com/
UPDATE: I published some Logisim implementations of the CPUs described in this book here.