No, you're not a horrible person! I'm sure you'll do everything you can to help him. I would talk to the vet about it when you take him there, and make sure there isn't a health issue underlying. Being moved around really stresses most cats out.
We were having numerous issues with one of our cats. He was urinating in the house, and was having episodes where his tail would "chase" him. After being medicated for a urine crystal build-up, and switching food to one that supports bladder health, it made a huge difference. However, we had just bought a townhouse and knew from previous moves that it really stresses him out.
After doing some research, we thought that trying
Feliway couldn't hurt. It's a plug-in diffuser that releases cat pheromones, and basically it makes your cat feel more relaxed/at home. It's expensive, and in Canada you can only get it from the vet, but I think in the states pet stores carry it. We ended up ordering some off of ebay, and plugged it in 2 days before we moved in. I really feel like it made his transition less stressful. It's supposed to decrease scratching, spraying, and evasive behaviour. Might be worth a try?