Background and Objective: As people age, cognitive function is an important indicator of health and mortality risk. An important factor affecting cognitive function is family members, especially spouse and children. This study investigated the cross-partner association between depressive symptoms and cognitive function. We further examined the cognitive performance of couples was affected by the frequency of contact with their adult children. Methods: The survey was conducted at the Veterans Medical Research Institute in the Veterans Health Service Medical Center. A total of 235 individuals volunteered for this study and provided their informed consent at enrollment. After the selection process, 96 couples participated in the present study. Participants completed a neuropsychological test battery including attention, language and related functions, visuospatial functions, memory, and frontal/executive functions. Symptoms of depression were assessed using the short version of the Geriatric Depression Scale. The number of contacts with children was categorized into ≥1 per month and <1 per month. Results: We found a significant association between depression and cognitive function between husbands and wives. Although the association was asymmetric: wives’ depressive symptoms were significantly associated with lower executive function tasks by their husbands, whereas the converse association was not significant. Similarly, within a couple, more contact with adult children (more than once a month) was associated with a significantly higher cognitive scores (particularly in visuospatial/executive functions) by husbands. These observed associations were independent of the individuals’ age, education, income, smoking status, alcohol consumption, engagement in moderate physical activity, and disease histories. Conclusions: Among older married couples, the cognitive functions of husbands tend to be influenced more by their wives’ mental health and degree of contact with their adult children. This infers that wives and offspring can act as buffer against the cognitive impairment of older married men.