I read this a long time ago so details are few. MC is a man who lives alone (in the woods I think). he is considered a wizard or something like that by his neighbors. What he really is a a refugee from one of Earth's colonies (Earth by this time has been more or less forgotten and reverted to a pastoral society. he moves every so often when people start to notice he does not age like every one else. At some point he is sought out and confronted by another visitor from the colonies.