Brunswick pastor afraid to return to work