Nonlocal adiabatic response of a localized system to local manipulations