I said this exact thing to a younger male coworker who was amazed at all the beautiful women I talked to at work, and that he said he couldn't be that cool talking to them, exclaiming, "Don't you just want to take them all home?"
"There's a difference between arousal and desire," I told him, which he didn't get; thankfully, I got called away and I dropped the subject.
Yes, I do work with very beautiful women, and yes some of them are very arousing; in other words, I find their physical appearance pleasing and attractive. And some of them are also very funny, sweet, sassy, smart, kind, caring, which also adds to the arousal factor (personally, I have never understood being turned on by someone you can't stand, but that's just me). But that does not mean I desire each and every one of them.
To me, desire isn't just "Hey, you're hot!" because to me, desire has a lot of other factors involved, such as age, compatibility, maturity, and trust. Because to me, I could be talking to someone who is gorgeous and funny and sweet, but if they remind me of my daughter (both in age and sayings) then that's a no-go for me. I could be talking to a woman my own age and if they give me sisterly vibes, that's also a hard pass (well, not hard at all, but you get what I mean LOL).
So I believe there's a difference between arousal and desire; yes, some people may not give 2 shits about the difference (or even acknowledge there is one), but that's what works for me :)