Reservoir heterogeneity significantly impacts the fluid flow behavior in porous media. In subsurface communities including hydrocarbon or geothermal recovery, geological storage of CO2 or H2 , and hydrology, transient pressure data are often used to infer the subsurface rock properties (e.g., permeability field) due to its ready availability and quick response. The accurate estimation of such properties is critical to accurately predict fluid flow in porous media. In this work, we propose a deep learning (DL) approach inspired by the Fast Marching Method (FMM), namely the Inversion Neural Network (INN), to inversely infer heterogeneous reservoir model parameters using transient pressure data. The forward model used to generate training data for the INN is established based on a semi-analytic asymptotic solution to the diffusivity equation using the diffusive time of flight (DTOF). The FMM solves the Eikonal equation for the DTOF and provides the partial derivative of pressure drop to the natural logarithm of time (hereafter pressure derivative). The pressure derivative data are then used to predict reservoir model parameters by the INN. In the homogeneous scenario, the INN architecture is a relatively simple fully connected neural network as a proof-of-concept to validate the feasibility, and it directly correlates the permeability value with the pressure derivative. In the heterogeneous scenario, as the heterogeneous permeability field is estimated based on sparse observational data of pressure derivatives, we adopt the convolutional neural network (CNN) to flexibly deal with the image-based properties, and leverage transfer learning to efficiently train a robust INN in the heterogeneous scenario. We first validated that INN can infer the homogeneous permeability fields through numerical experiments by testing root-mean-square-error (RMSE) around 1.086 𝑚𝑑. Inspired by that, observational data with different sparsity are used to train convolutional INN and predict heterogeneous permeability fields. As the number of observational locations (𝑛𝑜𝑏𝑠) increases from 3 × 3 to 48 × 48, the testing RMSE in heterogeneous scenarios decreases from 187.510 𝑚𝑑 to 18.080 𝑚𝑑. Besides, we found that transfer learning significantly improves the predictive accuracy at low 𝑛𝑜𝑏𝑠, with relative error decreased by 17.25%. Finally, noisy pressure derivative data under different 𝑛𝑜𝑏𝑠 are used to history match the heterogeneous reservoir model, and INN can infer the permeability field with a low error of 9.6% at 𝑛𝑜𝑏𝑠 = 7 × 7. Without an iterative procedure for parameter estimation, the INN demonstrates to perform permeability inversion with CPU time in the magnitude of 9.2 × 10−4 s on a single GPU (NVIDIA Quadro P2200). The FMM-inspired INN sets up the basis for the accurate characterization of reservoir model heterogeneity by inverting pressure derivative data with both decent predictive accuracy and computational efficiency.