Oh.. the poor old man.
Luckily you had the solution in your hands (or atleast in your stomach). This is how I fixed it:
1. goto the old man (this make sure I get the right one)
2. goto the gem (so I put the old man in the right place)
3. summon the old man (this gets the right old man)
4. give the gem to the old man
Done :-)
xiii coded the gem.. so Im not sure how vicious it is.. I hope you had fun not frustration.