Rustdoc takes 8.5 GB of memory to document stm32h7 #79103

jyn514 · 2020-11-16T15:40:56Z

Similar to but not the same as #78761: rustdoc uses an inordinate amount of memory to document large crates.

heaptrack_gui shows almost a third of the memory being allocated in get_blanket_impls:

However, the peak usage is actually in check_mod_item_types, which rustdoc can't do much about:

Rustdoc should work improving on both the total memory usage and peak memory usage.

This is meant as a tracking issue, since there are issues other than get_blanket_impls causing high memory use, such as #76382. There is a Zulip stream for technical discussion.

cc @adamgreig

The text was updated successfully, but these errors were encountered:

jyn514 · 2020-11-16T15:42:31Z

This is causing docs.rs to have to periodically bump the sandbox limits when documenting stm32 crates: rust-lang/docs.rs#1179.

camelid · 2020-12-11T22:31:48Z

P-medium: it's annoying that it takes so much memory, but it's not like you can't document it. And it's being worked on.

jyn514 · 2020-12-11T22:53:51Z

I ran Rustdoc under DHAT and found that a good tenth of the allocations come from get_blanket_impls alone:

rust/src/librustdoc/clean/blanket_impl.rs

Line 130 in 2225ee1

items: self

. ~~I'm not sure how to upload the full 81 MB file to GitHub, but feel free to ping me here or on Zulip, I'll keep a copy around~~ Here's a link to Google Drive with the full file. Unfortunately DHAT takes a full 50 minutes to run rustdoc on stm32 so it's a pain to collect on your own.

jyn514 · 2020-12-11T23:17:56Z

Another 7% just storing the structs themselves:

rust/src/librustdoc/passes/collect_trait_impls.rs

Line 128 in 2225ee1

self.impls.extend(get_auto_trait_and_blanket_impls(

. Maybe I should be asking just how many impls there are?

[rustdoc] Calculate span information on demand instead of storing it ahead of time This brings `size_of<clean::types::Span>()` down from over 100 bytes (!!) to only 12, the same as rustc. It brings `Item` down even more, from `784` to `680`. TODO: I need to figure out how to do this for the JSON backend too. That uses `From` impls everywhere, which don't allow passing in the `Session` as an argument. `@P1n3appl3,` `@tmandry,` maybe one of you have ideas? Helps with rust-lang#79103

[rustdoc] Calculate span information on demand instead of storing it ahead of time This brings `size_of<clean::types::Span>()` down from over 100 bytes (!!) to only 12, the same as rustc. It brings `Item` down even more, from `784` to `680`. ~~TODO: I need to figure out how to do this for the JSON backend too. That uses `From` impls everywhere, which don't allow passing in the `Session` as an argument. `@P1n3appl3,` `@tmandry,` maybe one of you have ideas?~~ Figured it out, fortunately only two functions needed to be changed. I like the `convert_x()` format better than `From` everywhere but I'm open to feedback. Helps with rust-lang#79103

[rustdoc] Box ItemKind to reduce the size of `Item` This brings the size of `Item` from ``` [src/librustdoc/lib.rs:103] std::mem::size_of::<Item>() = 680 ``` to ``` [src/librustdoc/lib.rs:103] std::mem::size_of::<Item>() = 280 ``` This is an alternative to rust-lang#79967; I don't think it makes sense to make both changes. Helps with rust-lang#79103.

[rustdoc] Box `ItemKind::Impl` to shrink the size of Item Helps with rust-lang#79103. Builds on rust-lang#79957 and should not be merged before. Eventually I want to calculate this info on demand (rust-lang#76382) but that turned out to be quite difficult so I'm leaving it for later and slapping on this short-term fix. This brings the size of Item and ItemKind from ``` [src/librustdoc/lib.rs:102] std::mem::size_of::<Item>() = 680 [src/librustdoc/lib.rs:102] std::mem::size_of::<ItemKind>() = 408 ``` to ``` [src/librustdoc/lib.rs:102] std::mem::size_of::<Item>() = 608 [src/librustdoc/lib.rs:102] std::mem::size_of::<ItemKind>() = 336 ``` Waiting to start a perf run until rust-lang#79957 lands.

Get rid of `clean::Deprecation` This brings the size of `item.deprecation` from 56 to 16 bytes. Helps with rust-lang#79103 and rust-lang#76382, in the same vein as rust-lang#79957. r? `@GuillaumeGomez`

[rustdoc] Switch to Symbol for item.name This decreases the size of `Item` from 680 to 616 bytes. It also does a lot less work since it no longer has to copy as much. Helps with rust-lang#79103. r? `@GuillaumeGomez`

Get rid of `clean::Deprecation` This brings the size of `item.deprecation` from 56 to 16 bytes. Helps with rust-lang#79103 and rust-lang#76382, in the same vein as rust-lang#79957. r? `@GuillaumeGomez`

[rustdoc] Switch to Symbol for item.name This decreases the size of `Item` from 680 to 616 bytes. It also does a lot less work since it no longer has to copy as much. Helps with rust-lang#79103. r? `@GuillaumeGomez`

jyn514 · 2020-12-16T03:02:00Z

Maybe I should be asking just how many impls there are?

+-------------------------------------------------+-----------+-----------------+----------+------------+
| Item                                            | Self time | % of total time | Time     | Item count |
+-------------------------------------------------+-----------+-----------------+----------+------------+
| collect-trait-impls                             | 33.31s    | 45.735          | 39.64s   | 1          |
+-------------------------------------------------+-----------+-----------------+----------+------------+
| render_html                                     | 23.69s    | 32.537          | 23.69s   | 1          |
+-------------------------------------------------+-----------+-----------------+----------+------------+
| <unknown>                                       | 6.81s     | 9.345           | 10.21s   | 2417746    |
+-------------------------------------------------+-----------+-----------------+----------+------------+
| clean_crate                                     | 2.85s     | 3.913           | 4.01s    | 1          |
+-------------------------------------------------+-----------+-----------------+----------+------------+
| collect-intra-doc-links                         | 918.47ms  | 1.261           | 970.03ms | 1          |
+-------------------------------------------------+-----------+-----------------+----------+------------+
| expand_crate                                    | 905.65ms  | 1.244           | 909.95ms | 1          |
+-------------------------------------------------+-----------+-----------------+----------+------------+
| build_extern_trait_impl                         | 796.51ms  | 1.094           | 1.38s    | 20080      |

The strange thing that most of the time is not spent on build_extern_trait_impl, it's somewhere else in collect_trait_impls.

jyn514 · 2020-12-16T03:10:00Z

Almost all the time in collect_trait_impls is being spent on synthetic impls, specifically

rust/src/librustdoc/passes/collect_trait_impls.rs

Line 19 in 4031f7b

let mut krate = synth.fold_crate(krate);

+-------------------------------------------------+-----------+-----------------+----------+------------+
| Item                                            | Self time | % of total time | Time     | Item count |
+-------------------------------------------------+-----------+-----------------+----------+------------+
| collect_synthetic_impls                         | 32.71s    | 44.967          | 37.56s   | 1          |

jyn514 · 2020-12-16T03:20:55Z

get_blanket_impls takes about 10x as long as get_synthetic_impls, but they're both pretty slow.

+-------------------------------------------------+-----------+-----------------+----------+------------+
| Item                                            | Self time | % of total time | Time     | Item count |
+-------------------------------------------------+-----------+-----------------+----------+------------+
| get_blanket_impls                               | 29.86s    | 41.053          | 34.48s   | 32176      |
+-------------------------------------------------+-----------+-----------------+----------+------------+
| render_html                                     | 23.18s    | 31.879          | 23.18s   | 1          |
+-------------------------------------------------+-----------+-----------------+----------+------------+
| <unknown>                                       | 6.92s     | 9.514           | 10.33s   | 2417746    |
+-------------------------------------------------+-----------+-----------------+----------+------------+
| get_auto_trait_impls                            | 3.42s     | 4.704           | 3.69s    | 32176      |
+-------------------------------------------------+-----------+-----------------+----------+------------+
| clean_crate                                     | 2.82s     | 3.871           | 3.98s    | 1          |
+-------------------------------------------------+-----------+-----------------+----------+------------+

Get rid of `clean::Deprecation` This brings the size of `item.deprecation` from 56 to 16 bytes. Helps with rust-lang#79103 and rust-lang#76382, in the same vein as rust-lang#79957. r? `@GuillaumeGomez`

Add more timing info to rustdoc This helped me confirm in rust-lang#79103 (comment) that get_blanket_impls is indeed what's taking all the time on stm32. r? `@Mark-Simulacrum`

[rustdoc] Calculate stability, const_stability, and deprecation on-demand Previously, they would always be calculated ahead of time, which bloated the size of `clean::Item`. Builds on rust-lang#80090 and should not be merged before. Helps with rust-lang#79103 and rust-lang#76382. cc rust-lang#80014 (comment) This brings `Item` down to 568 bytes, down from 616.

Remove `DefPath` from `Visibility` and calculate it on demand Depends on rust-lang#80090 and should not be merged before. Helps with rust-lang#79103 and rust-lang#76382. cc rust-lang#80014 (comment) - `@nnethercote` I figured it out! It was simpler than I expected :) This brings the size of `clean::Visibility` down from 40 bytes to 8. Note that this does *not* remove `clean::Visibility`, even though it's now basically the same as `ty::Visibility`, because the `Invsible` variant means something different from `Inherited` and I thought it would be be confusing to merge the two. See the new comments on `impl Clean for ty::Visibility` for details.

…meGomez [rustdoc] Calculate stability, const_stability, and deprecation on-demand Previously, they would always be calculated ahead of time, which bloated the size of `clean::Item`. Builds on rust-lang#80090 and should not be merged before. Helps with rust-lang#79103 and rust-lang#76382. cc rust-lang#80014 (comment) This brings `Item` down to 568 bytes, down from 616.

…umeGomez Remove `DefPath` from `Visibility` and calculate it on demand Depends on rust-lang#80090 and should not be merged before. Helps with rust-lang#79103 and rust-lang#76382. cc rust-lang#80014 (comment) - `@nnethercote` I figured it out! It was simpler than I expected :) This brings the size of `clean::Visibility` down from 40 bytes to 8. Note that this does *not* remove `clean::Visibility`, even though it's now basically the same as `ty::Visibility`, because the `Invsible` variant means something different from `Inherited` and I thought it would be be confusing to merge the two. See the new comments on `impl Clean for ty::Visibility` for details.

[rustdoc] Box ItemKind to reduce the size of `Item` This brings the size of `Item` from ``` [src/librustdoc/lib.rs:103] std::mem::size_of::<Item>() = 536 ``` to ``` [src/librustdoc/lib.rs:103] std::mem::size_of::<Item>() = 136 ``` This is an alternative to rust-lang#79967; I don't think it makes sense to make both changes. Helps with rust-lang#79103.

jyn514 added T-rustdoc I-compilemem C-bug C-tracking-issue labels Nov 16, 2020

jyn514 added T-compiler I-prioritize labels Nov 16, 2020

jyn514 mentioned this issue Dec 1, 2020

SIGKILL (OOM?) for documenting web-sys 0.3.46 rust-lang/docs.rs#1197

Closed

camelid added P-medium and removed I-prioritize labels Dec 11, 2020

jonas-schievink added the WG-embedded label Dec 11, 2020

jyn514 mentioned this issue Dec 12, 2020

[rustdoc] Calculate span information on demand instead of storing it ahead of time #79957

Merged

jyn514 mentioned this issue Dec 12, 2020

[rustdoc] Box `ItemKind::Impl` to shrink the size of Item #79967

Closed

This was referenced Dec 12, 2020

[rustdoc] Box some more ItemKind variants #79975

Closed

[rustdoc] Box ItemKind to reduce the size of `Item` #80014

Merged

This was referenced Dec 14, 2020

Get rid of `clean::Deprecation` #80041

Merged

[rustdoc] Switch to Symbol for item.name #80044

Merged

This was referenced Dec 16, 2020

Add more timing info to rustdoc #80071

Merged

[rustdoc] Calculate stability, const_stability, and deprecation on-demand #80095

Merged

Remove `DefPath` from `Visibility` and calculate it on demand #80099

Merged

jyn514 mentioned this issue Dec 29, 2020

Serialize incr comp structures to file via fixed-size buffer #80463

Open

rust-lang / rust

Rustdoc takes 8.5 GB of memory to document stm32h7 #79103

Rustdoc takes 8.5 GB of memory to document stm32h7 #79103

jyn514 commented Nov 16, 2020

jyn514 commented Nov 16, 2020

camelid commented Dec 11, 2020

jyn514 commented Dec 11, 2020 •

edited

jyn514 commented Dec 11, 2020

jyn514 commented Dec 16, 2020

jyn514 commented Dec 16, 2020

jyn514 commented Dec 16, 2020

rust-lang / rust

Rustdoc takes 8.5 GB of memory to document stm32h7 #79103

Rustdoc takes 8.5 GB of memory to document stm32h7 #79103

Comments

jyn514 commented Nov 16, 2020

jyn514 commented Nov 16, 2020

camelid commented Dec 11, 2020

jyn514 commented Dec 11, 2020 • edited

jyn514 commented Dec 11, 2020

jyn514 commented Dec 16, 2020

jyn514 commented Dec 16, 2020

jyn514 commented Dec 16, 2020

jyn514 commented Dec 11, 2020 •

edited