Just 66% of men were employed or actively seeking a job as of April, a 20-year low from 73% in 2006.