Gaff is British and Irish slang for home or house. "Come back to my gaff" means come to my place. Not concerning, just informal vocabulary. Has nothing to do with the old sense of a fishing hook or theatre slang. If your kid uses it constantly, ask them where they first heard it, not to police their speech but to understand which online community they are in.