James Francis Duffy (August 9, 1920 – November 29, 2003) was an American professional baseball umpire and a professional basketball referee. He worked in the American League from 1951 to 1953. Duffy umpired 448 major league games in his three-year career.