Does the Constitution Follow the Flag?

The Bush Administration has notoriously argued that detainees at Guantanamo do not enjoy constitutional rights because they are held outside American borders. But where do rules about territorial legal limits such as this one come from? Why does geography make a difference for what legal rules apply? Most people intuitively understand that location affects constitutional rights, but the legal and political basis for territorial jurisdiction is poorly understood.