When can the IRS take your passport for unpaid taxes, and does that prevent you from international travel? Yes, the IRS has the power to take many actions to collect unpaid US taxes including the ability to take away your passport. At present, you must owe the IRS $50,000 or...