The Compact C Type Format in the GNU toolchain [LWN.net]

The Compact C Type Format in the GNU toolchain

Posted Aug 7, 2019 15:53 UTC (Wed) by luto (subscriber, #39314)
In reply to: The Compact C Type Format in the GNU toolchain by roc
Parent article: The Compact C Type Format in the GNU toolchain

IMO what we need for locating debuginfo is a standard API by which a debugger can ask the distro to locate a debug info file. Distros patches to gdb are annoying, and they also preclude things other than gdb from reliably finding debug info files.

I would love for the kernel to be able to drop something in /usr/lib/debuginfo.d/find_vdso_debuginfo.sh, for example.

to post comments

The Compact C Type Format in the GNU toolchain

Posted Aug 7, 2019 15:59 UTC (Wed) by fuhchee (guest, #40059) [Link] (15 responses)

Some of us are working on just such a thing, and will present our progress at the GNU Cauldron in Montreal next month.

https://sourceware.org/git/?p=elfutils.git;a=tree;f=dbgse...

The Compact C Type Format in the GNU toolchain

Posted Aug 7, 2019 16:34 UTC (Wed) by luto (subscriber, #39314) [Link] (3 responses)

I peeked a bit. Will this also handle debuginfo files that are directly installed on the filesystem?

The Compact C Type Format in the GNU toolchain

Posted Aug 7, 2019 16:42 UTC (Wed) by fuhchee (guest, #40059) [Link] (2 responses)

Yup!

% dbgserver -F /path/to/base/directory

should find executables / debuginfo / corresponding sources

The Compact C Type Format in the GNU toolchain

Posted Aug 7, 2019 16:53 UTC (Wed) by luto (subscriber, #39314) [Link] (1 responses)

How does this ending up working on a normal desktop or server? Is the idea that there would be a systemwide dbgserver instance, perhaps socket activated, or maybe several instances?

As an admin, I would much rather *not* have a systemwide daemon for this, since that implies a path by which one user can attempt to attack another user or the system as a whole. I'd rather if each user could, on demand, start up their own instance of whatever libraries and programs are needed to make debugging work. Nothing here should require any form of privilege.

The Compact C Type Format in the GNU toolchain

Posted Aug 7, 2019 17:47 UTC (Wed) by fuhchee (guest, #40059) [Link]

> Is the idea that there would be a systemwide dbgserver instance, perhaps socket activated, or maybe several instances?

Any of the above.

>As an admin, I would much rather *not* have a systemwide daemon for this, since that implies a path by which one user can attempt to attack another user or the system as a whole.

Fair enough, though DoS is perhaps the worst of the possible attacks.

> Nothing here should require any form of privilege.

Right.

The Compact C Type Format in the GNU toolchain

Posted Aug 7, 2019 18:05 UTC (Wed) by madscientist (subscriber, #16861) [Link] (6 responses)

I hope that consideration of non-packaged / generic sources for debuginfo is being considered / allowed for. I've really wanted to start stripping out debuginfo from our binaries on my development team since they're so huge, but we don't create RPM or DEB files. When I propose using external debuginfo people are not happy about the developer overhead (even scripted) so we've never done it.

If it were possible to easily "register" debuginfo files created through Jenkins or some other build service without having to turn them into distro packages, then allow tools (i.e., GDB) to download them more or less invisibly when needed, that would be really nice.

Also is this limited to just debuginfo files?

The other big problem we have is cores being generated on remote systems which are using system libraries other than the local ones: in this situation we need to obtain the remote system's libc.so and other necessary system libraries. It would be really, really nice if we could register shared libraries from different systems, perhaps indexed via a hash of some kind, then have GDB automatically download them as well.

Of course, before this can be done we need to ensure that the core file contains enough information about the shared libraries to perform the lookup, which I doubt it does today, so this is requires more work in other places... however it would be good if the design of this service was able to be extended in this way in the future if/when it becomes feasible. For example in our system we use Google coredumper library rather than the kernel to dump cores and this allows us to add "notes" into the generated core file, so we could take advantage of this without Linux kernel support.

The Compact C Type Format in the GNU toolchain

Posted Aug 7, 2019 19:18 UTC (Wed) by fuhchee (guest, #40059) [Link] (5 responses)

Compiling buildids into your binaries is enough for this widget to find system shared libraries & their debuginfo (and possibly their sources), and serve them to a remote debugger.

It seems like we're all thinking roughly alike. Exciting times!

The Compact C Type Format in the GNU toolchain

Posted Aug 7, 2019 19:41 UTC (Wed) by madscientist (subscriber, #16861) [Link] (4 responses)

Hm... we must be talking about something different. Let be more clear.

I compile a program on my build system (I use a sysroot to ensure that it links against sufficiently older system libraries that it can run "anywhere"). I send my program out to run tests some other system running some random distribution completely different than the one it was built on, which is using a different GNU libc, etc. Maybe Travis, or AWS, or just a local test farm.

It fails and a core is generated. To debug that core I need my program, the debuginfo from my program (if the program is stripped), the core file, and the system libraries from the system it was running on when the core is generated.

I can't see any way that a buildid compiled into my binary can be sufficient to retrieve the runtime system libraries.

The Compact C Type Format in the GNU toolchain

Posted Aug 7, 2019 19:52 UTC (Wed) by fuhchee (guest, #40059) [Link] (3 responses)

The runtime system shared libraries have their own buildids, and the relevant ELF note sections should show up in the core dump. From those buildids, the relevant binaries / debuginfo can be found.

The Compact C Type Format in the GNU toolchain

Posted Aug 7, 2019 20:22 UTC (Wed) by madscientist (subscriber, #16861) [Link] (2 responses)

I see. So dbgserver_find_executable() is intended to be used with shared libs as well? Or is this part not quite complete?

I think it would also be helpful if the client interface provided separate lookup and download methods rather than forcing them to both be a single method (there can be a simplified "do both" method as well if wanted). I can easily imagine situations where we want to know whether a given buildid exists on the server without actually downloading it.

For example, suppose I have a suite of test servers running random environments; during test runs a core is generated. I want to know if the program under test and/or system libraries for this system already exist in the debug server or not: I just want to look them up but not download them. If they don't exist perhaps I'll include them along with the core file when I bundle up the build results. If they do exist I don't need to add them.

Or perhaps I have an automated way for the test system to upload binaries and/or system libraries that aren't already on the debug server (I understand that upload is not in scope for this project and would need some other process) but I don't want to bother uploading things that I already have so I need to be able to check.

A simple program that uses the client interface to look up and/or download files would be very useful, as an example if nothing else (and probably for people who would like to add scripting to systems where it's not so simple to recode them to use it).

Cheers!

The Compact C Type Format in the GNU toolchain

Posted Aug 7, 2019 21:09 UTC (Wed) by fuhchee (guest, #40059) [Link] (1 responses)

> I see. So dbgserver_find_executable() is intended to be used with shared libs as well?

Yes.

> I think it would also be helpful if the client interface provided separate lookup and download methods

Will consider that ... though there may be better ways to service the needs you outline. Deduplication at upload time should be easy too. Re. optimizing packaging of core dumps ... not sure how much sense that makes. The core dump recipient could consult the same debuginfo servers too; or you could preemptively package all the files. Will think on it more.

> A simple program that uses the client interface to look up and/or download files would be very useful

It just appeared in the repo! We employ only the most talented psychics and keyboard monks.

The Compact C Type Format in the GNU toolchain

Posted Aug 7, 2019 21:29 UTC (Wed) by madscientist (subscriber, #16861) [Link]

> Deduplication at upload time should be easy too

If you mean deduplication by the server that's probably helpful but it's a lot of wasted effort to upload 10's or 100's of MB of libraries, binaries, etc., only to have it tossed on the floor as duplicate. Consider a build farm with 200 systems, which are upgraded via apt-get update or whatever at random intervals so they have different system libraries, different program instances, etc... having every system upload all its files for every core even though the system libraries might only change once every few weeks or less seems like overkill.

> Re. optimizing packaging of core dumps ... not sure how much sense that makes. The core dump recipient could consult the same debuginfo servers too; or you could preemptively package all the files.

For this I wasn't thinking that the dbgserver code would do that, I was thinking about scripting that users are using with their test clients to bundle results of failures so they can be uploaded to a test server for further investigation. Our current scripting already preemptively packages all the files: what I'd like to be able to do is detect when some/all of these items are not needed and skip that to reduce the size of uploaded artifacts.

When you're talking about moving content into/out of AWS or other cloud providers, the amount of data sent over the network directly equates to $$ spent and reducing it is always welcome.

Thanks for working on this, it'll be very cool!

The Compact C Type Format in the GNU toolchain

Posted Aug 7, 2019 22:01 UTC (Wed) by nix (subscriber, #2304) [Link] (1 responses)

Ooh this is very nice, very nice indeed.

I wonder if it can handle more than just debuginfo... musing about having libctf automatically launch dbgserver queries for missing CTF sections now -- so people can have separated CTF if they really want *and* it is as if it were always present. Best of all worlds! For that matter they can also do both -- perhaps an option at CTF generation time which automatically emits separated CTF *if* its size passes some threshold, or some percentage threshold of the total binary size, or the .text size, or something. Of course then you'd have to arrange for the dbgserver to see it, but presumably whatever method is used for separated debuginfo would work for this too.

The Compact C Type Format in the GNU toolchain

Posted Aug 8, 2019 12:28 UTC (Thu) by fuhchee (guest, #40059) [Link]

> I wonder if it can handle more than just debuginfo... musing about having libctf automatically launch dbgserver queries for missing CTF sections now -- so people can have separated CTF if they really want

It should be a small amount of extra effort to extend it sideways to a 'ctf' sibling to 'debuginfo'.

The Compact C Type Format in the GNU toolchain

Posted Aug 7, 2019 22:26 UTC (Wed) by thoughtpolice (subscriber, #87455) [Link] (1 responses)

This is very interesting. We have a tool for NixOS that basically does except it is based on a FUSE filesystem and a content addressable packaging system: https://github.com/edolstra/dwarffs

Essentially, every version of every package has a unique hash. We build a reverse mapping from the buildids of the binaries in a package to its unique hash, and upload that metadata along with the package to the package server. We then patch GDB (and elfutils) to look in a specific directory for debug info. This directory is a FUSE filesystem, and when any tool tries to look in `.build-id/...` for the debug info -- it does a query to the package server, obtaining the unique package ID containing the symbols, and transparently installs them through the package manager. It is effectively a version of Microsoft Symbol Server, which is basically what people want, from what I can tell...

Perhaps we could replace dwarffs with something like dbgserver, however. Or integrate them so there's a single UX. We could for instance, perhaps replace the client tooling with a separate "backend" for our case, and the tools can all just work around that instead...

The Compact C Type Format in the GNU toolchain

Posted Aug 7, 2019 23:25 UTC (Wed) by fuhchee (guest, #40059) [Link]

Yup, was aware of your server! We wanted something more http flavoured and a little more distro-independent. Joining forces would of course be wonderful!