Kartick Chandra Pyne, Refugee family