“Hi, its not convenient for me right now, can I have your home phone number, and I’ll call you back shortly”

and, when they ask for any information from me:

“I’m sorry, I can’t verify who you are – you called me out of the blue. You understand that I need to know who I’m talking to before I provide personal details. Could I trouble you for your contact number and I’ll call you back to verify who you are?”. I usually forget where I may or may not have written down their number.

Both work like a charm.