AWS Inferentia features a large amount of on-chip memory which can be used for caching large models, instead of storing them off-chip. This has a significant impact in reducing inference latency since Inferentia’s processing cores, called Neuron Cores, have high-speed access to models that are stored in on-chip memory and are not limited by the off-chip memory bandwidth.