Background: Co-residential family members are still the main caregivers of informal care in Japan. Previous studies showed that the intensity of the caregiving is much greater for co-residential caregiving compared to extra-residential caregiving, and that those providing high-intensity caregiving (20 hours or more per week) were at twice the risk of psychological distress as noncaregivers. Objectives: To find out the factors that determine the hours spent on caregiving in Japan. I also analyze the effect of the hours per week of caregiving on the mental health of caregivers. Data and methods: I use 5-years longitudinal data (2005–2009) taken from the Longitudinal Survey of Middle-aged and Elderly Persons by the Japanese Ministry of Health, Labor and Welfare. The respondents who are the subjects were 50-59 years old in 2005. The respondents scored each item of K6 variables on a 5-point scale from 0 (never) to 4 (always true), with a high score indicating a severe psychosocial impairment. I use the dynamic random-effects probit model to reveal the gender difference in the probability of continuation of caregiving. An instrumental variables (IV) estimation of panel data is employed to examine the determinants of the hours spent on caregiving, because most female caregivers appeared to adjust their work hours due to caregiving. I use the lagged values of hours worked and income per month as instrumental variables when estimating the regression function of the hours per week of caregiving by the females. Results: The dynamic random-effects probit model for caregivers with work indicated that caregivers with higher reservation wages do not quit caregiving at the 5% significance level. The averaged parameters of the lagged caregiving were 0.460 and 0.188 in the females and males, 5 respectively. The gender difference in the probability of continuation of caregiving was large. The estimation result of the fixed-effects IV model in females showed that the number of hours spent on caregiving was greater when the recipient was the spouse’s mother. In contrast, the number was lower when high-intensity caregiving was provided in the previous period. The random-effects IV model in females suggested that a high number of hours per week of caregiving had a positive effect on the poor mental health of caregivers. Female caregivers sending living expenses for nonhousemates of the caregivers and male caregivers with heart disease tended to be in poor mental health. Having difficulty in daily life activities had a positive effect on poor mental health in females and in males. Policy implications: There were gender differences in the main determinants of the mental health of the caregivers. The longer hours of caregiving caused a higher prevalence of mental health problems among female caregivers. Improving female caregivers’ mental health by reducing their hours per week of caregiving is a very important health policy.