Investigating the Nature of Late-time High-energy GRB Emission through Joint Fermi/Swift Observations