The Compact C Type Format in the GNU toolchain [LWN.net]

The Compact C Type Format in the GNU toolchain

Posted Aug 7, 2019 21:55 UTC (Wed) by nix (subscriber, #2304)
In reply to: The Compact C Type Format in the GNU toolchain by mjw
Parent article: The Compact C Type Format in the GNU toolchain

Sorry, RPM uses elfutils eu-strip, which will not have special magic to treat .ctf sections specially.

I guess that means CTF will always be stripped out of RPMs, and libctf and the CTF format by extension will be useless on RPM systems. This seems unfortunate. Is it really so hard to mark .ctf sections as not stripped? If it takes more than a couple of lines, something seems to me to be wrong.

(This is not the only tool missing support right now, of course: gold can't link CTF sections either. But I plan to add that and I did also plan to submit changes to elfutils to stop eu-strip throwing the section out. I'm rather unhappy to discover that this is pre-emptively rejected.)

But I do like CTF and I do hope it will become the default one day. Not to replace DWARF (it should be a companion to that), but to replace .gnu_debugdata [***]. Which is used by various tools now to have an "extra symbol table".

That won't work, I'm afraid. CTF does not contain a symbol table, since that would be a waste of space since ELF already has one. Instead, it relies on the ELF symtab. Its function and data object sections are 1:1 ordered in the same order as the ELF symtab (basically, you traverse the ELF symtab and whenever you pass another function symbol, you match it to a function info section entry: whenever you pass another data symbol, you match it to another data object section entry). This saves quite a lot of space: data object section entries in particular are only four bytes each (one type ID).

(To deal with the problems of dynamic symbol tables getting stripped out of binaries, Solaris defined .ldynsym, which appears to be much what .gnu_debugdata is, only it's just a symbol table rather than a whole LZMA-compressed ELF object containing a symbol table.)

Plus of course there's not much chance of CTF becoming the default if you insist on stripping it out of executables so nothing that needs it can ever find it. ;)

to post comments

The Compact C Type Format in the GNU toolchain

Posted Aug 7, 2019 22:28 UTC (Wed) by mjw (subscriber, #16740) [Link] (5 responses)

> I guess that means CTF will always be stripped out of RPMs, and libctf and the CTF format by extension will be useless on RPM systems. This seems unfortunate. Is it really so hard to mark .ctf sections as not stripped? If it takes more than a couple of lines, something seems to me to be wrong.

It shouldn't be hard to keep it, if a package or distro decides that is the thing they want.
For example rust packages do something like:
%global _find_debuginfo_opts --keep-section .rustc

So all we need to do is define some macro that packages can set for find-debuginfo.sh to do "the right thing" and then a package or distro can decide to make that the default.

> I did also plan to submit changes to elfutils to stop eu-strip throwing the section out. I'm rather unhappy to discover that this is pre-emptively rejected.

That is not my intention. Note that I am a not a native English speaker. My apologies if I seem to come over as negative.

> CTF does not contain a symbol table, since that would be a waste of space since ELF already has one. Instead, it relies on the ELF symtab. Its function and data object sections are 1:1 ordered in the same order as the ELF symtab (basically, you traverse the ELF symtab and whenever you pass another function symbol, you match it to a function info section entry: whenever you pass another data symbol, you match it to another data object section entry). This saves quite a lot of space: data object section entries in particular are only four bytes each (one type ID).
>
> (To deal with the problems of dynamic symbol tables getting stripped out of binaries, Solaris defined .ldynsym, which appears to be much what .gnu_debugdata is, only it's just a symbol table rather than a whole LZMA-compressed ELF object containing a symbol table.)

OK. So how do you deal with .symtab being stripped away by default then?
Would it be an idea to adopt the .ldynsym from Solaris?
.gnu_debugdata was defined before we had compressed ELF sections in the standard.
Now that we have it maybe we should make .symtab a compressed section?
https://gnu.wildebeest.org/blog/mjw/2016/01/13/elf-libelf...

> Plus of course there's not much chance of CTF becoming the default if you insist on stripping it out of executables so nothing that needs it can ever find it. ;)

Really, I don't understand why you think that is my intention. I might not fully understand all details yet. But I am actually interested in making CTF into something useful.

Will you be at the GNU Tools Cauldron in Montréal, Canada next week?
It might be easier to talk some ideas over in person.
https://gcc.gnu.org/wiki/cauldron2019

The Compact C Type Format in the GNU toolchain

Posted Aug 7, 2019 22:32 UTC (Wed) by mjw (subscriber, #16740) [Link] (1 responses)

> Will you be at the GNU Tools Cauldron in Montréal, Canada next week?

Sorry, next month. (September 12 to 15, 2019)

The Compact C Type Format in the GNU toolchain

Posted Aug 9, 2019 11:33 UTC (Fri) by nix (subscriber, #2304) [Link]

Ah. I thought the timing you gave was strange for the Cauldron! That *overlaps* with LPC so in the absence of a teleporter or military jet to get from Lisbon to Montreal in no time... :/

The Compact C Type Format in the GNU toolchain

Posted Aug 9, 2019 0:51 UTC (Fri) by himi (subscriber, #340) [Link] (1 responses)

Sorry, RPM uses elfutils eu-strip, which will not have special magic to treat .ctf sections specially.

This can be read as meaning "no version of eu-strip will ever have the special magic", rather than what I believe you meant: "any eu-strip you find in the world right now will not have the necessary special magic".

The Compact C Type Format in the GNU toolchain

Posted Aug 9, 2019 11:21 UTC (Fri) by nix (subscriber, #2304) [Link]

Oh. Yes, that is how I parsed it. I completely understand that *existing* strip tools will strip this out. This just means that if you upgrade binutils to a CTF-generating version, you'd have to upgrade elfutils to one that doesn't strip it out as well. Given that you're already stuck having to upgrade the compiler in synchrony too to get this stuff to work, adding one extra package doesn't sound like an intolerable administrative burden. (I hope.)

The Compact C Type Format in the GNU toolchain

Posted Aug 9, 2019 11:32 UTC (Fri) by nix (subscriber, #2304) [Link]

> %global _find_debuginfo_opts --keep-section .rustc

... now why didn't I think of that? Probably because when I was doing this back when I had *multiple* sections to deal with, with names like .ctf.*, so telling other things what the sections were called this time would have been quite painful. We have our own internal container format now precisely to avoid this sort of problem, so we could use this quite well.

> That is not my intention. Note that I am a not a native English speaker. My apologies if I seem to come over as negative.

Sorry, I completely misparsed your sentence! (See my comment a couple of hops down). Phew, that had me panicking a bit for a moment. :)

> Now that we have it maybe we should make .symtab a compressed section?

Compressed sections in GNU ld at least seem a bit ad hoc. I think you'd need to do quite a lot of work to bfd_elf_final_link and environs to make it possible to have allocated sections that other sections depend upon that are also compressed: every existing section with content that changes after layout time (earlier in bfd_elf_final_link than strtab / symtab layout time) either has unchanging size or is non-allocated (and even there, there are fairly dreadful hacks around .zdebug, which I'm afraid I made a little bit worse with .ctf :) ).

I'm not sure .symtab would compress terribly well, either -- it has a lot of fields with "ID-like" content that only appears once, and thus compresses rather badly. (CTF goes to some lengths to avoid content like this for just that reason). The strtab would certainly compress better, but I can see why you don't compress it -- you don't want to impose decompression costs on the whole strtab on every execution.

> Really, I don't understand why you think that is my intention.

A really terrible misparsing of an ambiguous sentence on my part, and you know how hard it is to find alternate meanings of a sentence when you've already fixated on one that is dreadful :) Sorry!

> Will you be at the GNU Tools Cauldron in Montréal, Canada next week?

Alas, I'm going to LPC instead. I'm spending the next two weeks listening to chamber music in the North York moors...