Drop dirty dbengine pages if disk cannot keep up #7777
Conversation
Manage this branch in SquashTest this branch here: https://mfunduldbengine-drop-dirty-pag-n9rmv.squash.io |
mfundul
force-pushed the
mfundul:dbengine-drop-dirty-pages
branch
from
b43998e
to
c1648af
mfundul
changed the title
|
I've started reviewing the code - but this will take a few days. Last time this code was changed we saw some subtle bugs so I need to read around it more this time. Conveniently I'm examining the dbengine code for the roadmap task anyway so there is some overlap... caller-graph: call-graph: |
…t invalidates pages when the disk cannot keep up with the flushing speed.
mfundul
force-pushed the
mfundul:dbengine-drop-dirty-pages
branch
from
b145560
to
1054a6e
| array[33] = (uint64_t)ctx->stats.flushing_errors; | ||
| array[34] = (uint64_t)global_flushing_errors; | ||
| assert(RRDENG_NR_STATS == 35); | ||
| array[33] = (uint64_t)ctx->stats.pg_cache_over_half_dirty_events; |
amoss
Jan 30, 2020
Contributor
There is a comment about that changing the indices can break user code. Does changing the use of index 33 from the number of flushing errors to the number of over_half_dirty events affect any user code?
mfundul
Jan 30, 2020
Author
Contributor
All corresponding statistics have been adjusted. The hazard of memory corruption is about the number of the actual statistics. Changing individual statistics only changes the semantics.
amoss
Jan 30, 2020
Contributor
Ah ha. I see that you have changed the number of the actual statistics as well...
|
this #7777 is lucky one we need to merge it asap |
|
I am approving. I can not find something that shouldn't work as expected. Steps:
|
* Introduce dirty page pressure handling in the dbengine page cache that invalidates pages when the disk cannot keep up with the flushing speed.
Summary
Fixes #7268
Fixes #7468
Component Name
database/engine
Additional Information
Introduce dirty page pressure handling in the dbengine page cache that invalidates pages when the disk cannot keep up with the flushing speed. Make sure progress is made by managing real threads instead of relying on libuv workers since the workers will be exhausted by internal I/O queuing.