Build artefacts aren't cached? [LWN.net]

Build artefacts aren't cached?

Posted Oct 28, 2024 21:40 UTC (Mon) by Karellen (subscriber, #67644)
Parent article: The performance of the Rust compiler

it is often the case that a build will get hung up waiting for one large, commonly-used crate to build

...

Li finished the talk with a few areas [...] that might be fruitful to investigate: [...] cached binary crates

I find it very surprising that this is not already the case.

to post comments

Build artefacts aren't cached?

Posted Oct 29, 2024 10:03 UTC (Tue) by Wol (subscriber, #4433) [Link] (8 responses)

Okay, I gather the way Rust works makes it hard to create object files, but would it be possible to cache things at the point you cease processing a source file?

Something along the lines of "pull in all include files, run md5 over the resulting artifact, and if there's a cache with that name that's the processed output. If there's no cache, run the 'source to IR' pass, plus any local optimisations, and dump that to cache".

Dunno whether it's practical, or how much it would save, but it would certainly save reprocessing the same source time and again, and if there's a decent bunch of optimisations you can run before you start linking with other files, it would save that too ...

Cheers,
Wol

Build artefacts aren't cached?

Posted Oct 29, 2024 11:46 UTC (Tue) by mathstuf (subscriber, #69389) [Link] (1 responses)

For that, `sccache` works fairly well. It only does source -> object caching, but another intermediate level of caching might be worth it; needs measurements to be sure though.

Build artefacts aren't cached?

Posted Oct 29, 2024 13:10 UTC (Tue) by intelfx (subscriber, #130118) [Link]

> For that, `sccache` works fairly well. It only does source -> object caching

sccache works on crate level, not source file level. That’s exactly the opposite of what GP wanted (and it’s far from perfect, at that).

Build artefacts aren't cached?

Posted Oct 29, 2024 14:31 UTC (Tue) by josh (subscriber, #17465) [Link] (5 responses)

Rust does something like this (but at an even more fine-grained level) when doing incremental compilation. And not just at a file level: it even avoids recompiling functions which haven't changed within files that have.

Build artefacts aren't cached?

Posted Oct 29, 2024 14:59 UTC (Tue) by Wol (subscriber, #4433) [Link] (4 responses)

Is that a well known thing then?

Sounds like that's almost what people are looking for when iterating through the development process.

Cheers,
Wol

Build artefacts aren't cached?

Posted Oct 30, 2024 16:09 UTC (Wed) by josh (subscriber, #17465) [Link] (3 responses)

Incremental is on by default in dev builds (which are the default). They're off by default in release builds, for various reasons. That's something we may want to reconsider.

Build artefacts aren't cached?

Posted Oct 30, 2024 16:31 UTC (Wed) by Wol (subscriber, #4433) [Link]

That actually makes a lot of sense.

So small changes while coding should compile pretty quickly.

Personally, I'd be a lot happier with "incremental=off" for release builds, simply because it gives an aura (maybe false?) of reproducibility. Or rather, it addresses the assumption that dev builds can't be reproducible "because".

Cheers,
Wol

Build artefacts aren't cached?

Posted Oct 30, 2024 16:39 UTC (Wed) by intelfx (subscriber, #130118) [Link] (1 responses)

> They're off by default in release builds, for various reasons. That's something we may want to reconsider.

Is there something that elaborates on these reasons?

Build artefacts aren't cached?

Posted Oct 31, 2024 9:26 UTC (Thu) by taladar (subscriber, #68407) [Link]

I believe the main reason is that some optimizations work better when they can consider the whole instead of just an incremental part.