Hi,
I'm a bit sorry, but you trapped on a sort of unmaintained documentation.

1. The 'mostly in memory' is only true if you choose the MMFiles-Storageengine. Its name origins from 'Memory Mapped files' - which is that you can map on disk files into memory.
   This also means, that if you access parts of the files where the kernel decided that its not part of the "hot set" it needs to keep buffered in RAM it will load it from that disk file. The query engine working on top of this doesn't notice anything about that - maybe except that its not as fast.

2. yes, if you go outside of your hotset, performance is not as good, resource usage increases.

3. the operating system kernel decides what to do here.

4. you can influence the resource usage of your collections. If it needs to be 'loaded' the index has to be built up in memory, which will have a look at all documents, and add references to them to its index.

5. by 'loading' a collection you have it available in RAM.

For the rocksdb storage engine we explain whats actually happening in this blog article:
https://www.arangodb.com/2019/03/small-steps-reduce-arangodb-resource-footprint/

Hi @dothebart ,

Is there any specific reason why the Disk Reads on ArangoDB will sudden spike up, is it due to data indexing from main memory to secondary storage?

Attached the CPU,Disk Reads and memory utilization of our read only arangodb

Disk reads will be almost constant through out the day, but suddenly it will spike drastically and becomes normal after some time

could you please explain this behavior

Do you also see high(er) write activity in that period? If yes, the reason could be the rocksdb compaction, which has to re-oranize your data.
The database itself will store writes / updates at the time they're executed as new documents. Once all their steps are brought together and slack space inside of the database files are filled this is the compaction. It happenes during off-hours.

its read only machine which we create daily by taking AMI backup of master db machine. so they no disk writes on this machine.

we observe this behavior daily, due to high disk reads(on read only machine) all queries getting queued up and shown as slow queries and we have restart the machine.

we have increased the IOPS from 1.5k to 3k though we are observing this behavior.

if it is due to compaction? is there any way to schedule it in off-load hours

well, if the master has had writes, it derives that state of its data structures, right?
Yes, if you suffer from no more available burst credits or so everything comes to a grinding halt.

if it is due to compaction? is there any way to schedule it in off-load hours

@dothebart can you update on this

I'm closing this since ArangoDB 3.4 is EOLed meanwhile.

The best way to observe what process is doing in such a situation is, to run gcore (please note that gdb 9 is required with recent arangods at least) and top to inspect which threads (they should have name) are actually using the CPU.

Please note that ArangoDB 3.7 is available.