Store SnapShot method does not buffer read the database which could be an issue if database size is large #273
Labels
Comments
Thanks -- it's a weakness, I can look into addressing. |
Hi again, Philip! |
Is it actually causing you an issue?
… On Feb 14, 2017, at 11:46 AM, Ivan ***@***.***> wrote:
Hi again, Philip!
Do you happen to know how to fix this problem? I guess it's a "bug" in hashicorp/raft implementation.
Because as far as I understood, hashicorp/raft doesn't call Snapshot() and Persist() concurrently, so I don't know of a way to communicate chunks of the database between Snapshot() and Persist() - do you have an idea how to do this?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.
|
I mean, are you actually encountering a problem in practice?
… On Feb 14, 2017, at 11:46 AM, Ivan ***@***.***> wrote:
Hi again, Philip!
Do you happen to know how to fix this problem? I guess it's a "bug" in hashicorp/raft implementation.
Because as far as I understood, hashicorp/raft doesn't call Snapshot() and Persist() concurrently, so I don't know of a way to communicate chunks of the database between Snapshot() and Persist() - do you have an idea how to do this?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.
|
otoolep, thanks a lot! I've sorted it out with the raft library authors - here |
Thank you @JeanGolang -- I will look into using the new interface exposed by the Raft package. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Hello Philip!
I think there is an issue with this part of the code of rqlite (store/store.go).
As far as I understood from your code it's a support function for creating a snapshot of the database. However, this implementation reads the whole database into memory, in your particular case in the variable fsm.database, err = s.Database(false), so if the size of your database is larger than RAM available, I guess the program would crash.
I guess it's better to read the database in buffers or send chunks of the database via a channel when writing it to the so called SnapshotSink.
Thanks!
The text was updated successfully, but these errors were encountered: