I think you might get more advice from the chihuahua chat/help threads. It would be OK in there because it does concern your chi! I don't know if everyone reads these threads.
As for me, it would really worry me, but it would also be hard to give up too soon. I guess you are going to have to be the alpha and watch them, then give a strong NO the minute stuff starts happening. I had the same problem with two big dogs once and did have to find a new home. I was so sad, but you can't risk anyone's life/health!