"I know Nicole has talked to Tom about them spending a few weeks straight with her and Keith later this fall and Tom wasn’t for it at all," another source was quoted, as saying.

"He told Nicole that if she and Keith want to spend more time with them, they should head out to L.A. He's not going to uproot their schooling or their activities," the source adds.

Another reason for Tom's obstinate behavior is believed to be Keith's substance abuse problems.

"Keith's substance abuse problems may play into it. Tom's more wary of him than he was going in," another source added.

But according to the report, the couple will not give up despite Tom's unwillingness.

"Nicole just plans to work on Tom a bit, have Keith show his stripes by maintaining his sober path and get Tom to loosen his grip. She's been excited about Isabella and Connor getting to know Keith. But really, they've only been together a handful of times. I think it's still a dream for Nicole to have her kids consider her and Keith family the way they do with Tom and Katie. They have to make up for lost time,"an insider said.