How do I manage a married man who has become very close to me and we both know we're attracted?

It has been a week since it became apparent that we are attracted to each other. We have been super sweet and endearing towards each other. He is uber sweet. This morning, he told me Someday I would lie beside you, watch u wake up and hear you say good morning. I've been bothered because he is married at 46 with kids and Im 29 neveer been married, no kids and just broke up with a boyfriend of 2 years. I told him while its too early to say anything ant us, i worreis me because i know if we get deeper than how we already are, i am bound to get hurt. He told me not to think much about what he said. I know it is wrong to even be with him. But when I tried not to respond to his sms's and calls, it was just difficult. I dont think I can let time pass without being with him-- even if i know it may be temporary... but its wrong... how should i approach this dillema? What positioning should i take? Im falling and i know it. With him i am secure about myself. I just know its him.