First let me point out that even though one has the Power of Attorney, he does not have to act pursuant to its authority. However, there can only be one valid Power of Attorney at a time so it would be unethical for the Attorney in Fact to refuse to act if necessary.  The legal way to relinquish one’s responsibilities is to file a Revocation of the Power of Attorney, signed by the Grantor of the POA.  Alternatively, the Grantor of the POA could file a new POA naming someone else and this would automatically revoke the current Attorney in Fact.  This has to be signed, notarized and filed in the same Register of Deeds office where the original Power of Attorney was filed. It is best to have an attorney prepare this so that one is sure it accomplishes the revocation.  Then, someone else can be appointed by a new Power of Attorney.