ListView and GridView + contentY performance

Setup:
ListView + QAbstractItemModel containing several hundred items
Scrollbar which modifies contentY parameter of ListView (based on http://bitbucket.org/gregschlom/qmlscrollbar/src/tip/ScrollBar.qml, only without the two buttons to go up and down)

Problem:
If contentY is within the cached area (ListView cacheBuffer), scrolling is instantaneous. However, the cache does not seem to be updated. By scrolling further in one direction, uncached elements will be accessed.
In this case, almost the entire cache has to be rebuilt at once. This is a time-consuming operation which causes the application to freeze. As a result, contentY cannot be used to scroll in a ListView manually. The user would have to wait every x items for a few seconds.
Performing bigger jumps (e.g. from one end to the other) seems to have an even greater impact on the performance.

It does seem strange that jumping in a ListView appears to be more time-consuming than creating the entire model in the first place.