[RFC] Adding SFrame support to llvm

SFrame (“simple frame”) is a new format for unwinding the stack. It is used inside the Linux Kernel and has good support in the GNU toolchain. It is more fully described here:

I have a prototype implementation for both assembly and linking nearing completion at GitHub - Sterling-Augustine/llvm-project at sframe

It needs a fair amount of polish before it can be merged, but I would like to get started on that process. I expect the first part to be contributed would be adding a description in BinaryFormat, then the ability to read .sframe sections in objdump.

I’m told I should consult with upstream before starting to send patches.

Does anyone have questions or comments?

6 Likes

This looks like a good addition to LLVM.

Please submit the PR for dumping support first, so you can write regression tests for the assembler bits using it.

There’s a spec for the binary format, but I don’t really see a spec for the assembler support. Are there not any assembler directives because we expect the assembler to synthesize the section from DWARF-unwind directives? Is there some way to enable .sframe without enabling .eh_frame or .debug_frame? Is there some assembler directive for enabling .sframe, so we don’t need to pass a command-line flag to the assembler?

Will definitely do the objdump portions first.

Assembler support has very minimal additions syntactically. .sframe sections are synthesized from the same cfi directives that translate to .eh_frame and .debug_frame. There is a new assembler flag “–gsframes”, which generates sframes unconditionally, and a new possible parameter to the .cfi_sections directive:

.cfi_sections [.eh_frame | .debug_frame | <target specific sections> | .sframe]

And from there it all happens without additional work.

Generating these is orthogonal to other unwind info sections. One could generate all three or four if one wanted. But they are all independent.

Hello everyone,

I am going to be working with @Sterling-Augustine on implementing this feature – mainly the dumping part. I’ve just sent the first PR defining the relevant constants and structures for the format. Please take a look.

regards,
Pavel

1 Like

I am not comfortable with upstreaming the current version of SFrame in its entirety due to significant ELF specification violations and design concerns that primarily affect linkers.
That said, since the binary utilities and assembler patches have essentially landed, I am fine with these non-linker components remaining.
The critical issues I’ve identified are largely linker-specific (~1000 new lines to lld/ELF, very significant for a new feature) and don’t affect the assembler or object file dumping support.

A future version that addresses the linker-related issues—particularly the ELF specification violations and linking/execution view separation—would be suitable for full LLVM integration including LLD support.
I have a more detailed write-up at Remarks on SFrame | MaskRay

In addition, overall, I believe this feature was upstreamed prematurely—without sufficient analysis of its benefits, which is unusual for a feature requiring code changes of this scale.

ELF Specification Violations

The most critical issue is section group compliance. The current design creates a monolithic .sframe section containing relocations to STB_LOCAL symbols in discardable sections.

When inline functions are deduplicated across translation units, linkers (including gold and lld) correctly reject this:

ld.lld: error: relocation refers to a symbol in a discarded section: .text._Z3foov

Relocatable linking behavior

Currently, Binutils enforces a single-element structure within each .sframe section, regardless of whether it resides in a relocatable object or final executable. This approach differs from DWARF sections, which support multiple concatenated elements, each with its own header and body.

This design choice stems from Linux kernel requirements, where kernel modules are relocatable files created with ld -r. The kernel’s SFrame support expects each module to contain a single indexed format for efficient runtime processing. Consequently, GNU ld merges all input .sframe sections into a single indexed element, even when producing relocatable files. This behavior deviates from standard relocatable linking conventions that suppress synthetic section finalization.

A future version should distinguish between linking and execution views:

  • Linking view: Assemblers produce a simpler format, omitting fields needed only for indexing
  • Linkers concatenate .sframe input sections by default
  • A new --sframe-index option enables linkers to build the full indexed format for executables and shared objects, similar to --gdb-index and --debug-names. To support the Linux kernel workflow, ld -r --sframe-index should also function properly.

Questioned Benefits

The advantages over alternatives remain unclear:

  • Frame pointer unwinding: The format comparison lacks evaluation of enhanced frame pointer unwinding with prologue/epilogue pattern detection
  • Hardware assistance: Upcoming features like x86 Shadow Stack and AArch64 Guarded Control Stack may significantly reduce the need for metadata-based unwinding
  • Practical deployment: System-wide profiling faces limitations with syscalls, BPF helpers, and JIT support (see Some thoughts [LWN.net])

While metadata-based unwinding has value, these alternatives and challenges need proper evaluation.

Required Changes Before Upstreaming

I recommend addressing the ELF violations and implementing a proper linking/execution view distinction before upstreaming. The next version should:

  • Generate individual .sframe sections with proper group membership
  • Figure out separate linking and execution views and add a linker option for merged .sframe index for ld -r

Review Process Concerns

I have concerns about how this patch was landed:

Initial landing (Sept 17): The patch was merged without review from regular MC contributors, despite substantially modifying MC files that I maintain and modify. I was not added as a reviewer, even though github should have made this automatic as I heavily modify these files.
The only review feedback explicitly stated “It’s better to have another review for the MC part,” acknowledging the MC portion needed review. No regular MC contributor approved the patch.

Reland (Sept 18): After the patch was reverted due to test failures, I provided some review comments. The patch was relanded within 15 hours without waiting for approval or allowing time to address the feedback.
For new features of this complexity, as a new MC contributor please select some regular MC reviewers and not just your colleagues. This ensures proper domain expertise is applied and maintains code quality.

tl;dr: Not supporting sframes in LLD would be very unfortunate and bad for the llvm project.

There are three issues; I’ll address each in turn:

How technically deficient is the format?

I don’t think anyone would argue that there are not improvements to be made. However, calling the monolithic section an “elf violation” here overstates the problems a great deal. Yes, it is a single-section that cannot be appended one after the other, and yes, it doesn’t follow the semantics of elf groups. In particular, a non-group section has relocations that point into discardable sections.

However, there is strong precedent within elf for this, especially within unwind info: Both .eh_frame and .debug_frame work this way. Here are the section headers and relevant relocations for a small C++ that contains a group for a discardable function. It includes both sframes and eh_frames, and the pattern is identical:

$ cat u.cpp
template<class T> class bar { public: void foo(const T t) { } };
void func() { bar<int> b; b.foo(6); }

augustine:~/tmp $ g++ -Wa,-gsframe u.cpp -c && objdump -h u.o -r
u.o:     file format elf64-x86-64

Sections:
Idx Name          Size      VMA               LMA               File off  Algn
  0 .group        00000008  0000000000000000  0000000000000000  00000040  2**2
...

  4 .text._ZN3barIiE3fooEi 0000000e  0000000000000000  0000000000000000  00000064  2**1
                  CONTENTS, ALLOC, LOAD, READONLY, CODE
...
                  CONTENTS, READONLY
  7 .eh_frame     00000058  0000000000000000  0000000000000000  000000a0  2**3
                  CONTENTS, ALLOC, LOAD, RELOC, READONLY, DATA
  8 .sframe       00000062  0000000000000000  0000000000000000  000000f8  2**3
...

RELOCATION RECORDS FOR [.eh_frame]:
OFFSET           TYPE              VALUE
0000000000000020 R_X86_64_PC32     .text
0000000000000040 R_X86_64_PC32     .text._ZN3barIiE3fooEi

RELOCATION RECORDS FOR [.sframe]:
OFFSET           TYPE              VALUE
000000000000001c R_X86_64_PC32     .text
0000000000000030 R_X86_64_PC32     .text._ZN3barIiE3fooEi

Clang generates a similar set of relocations for .eh_frame:
RELOCATION RECORDS FOR [.eh_frame]:
OFFSET           TYPE              VALUE
0000000000000020 R_X86_64_PC32     .text
0000000000000040 R_X86_64_PC32     .text._ZN3barIiE3fooEi
0000000000000060 R_X86_64_PC32     .text+0x0000000000000020

All four open-source linkers, gnu-ld, gnu-gold, lld, and mold handle .eh_frame without even a comment about how this would be better handled in groups. And all four properly discard the relevant portion of .eh_frame when the group sections are discarded. It’s as if the reference to the discarded sections never existed, and the reference to the kept section is sufficient for unwinding.

The logic to do this is not difficult for formats designed for it. SFrame is designed for it, just like eh_frame.

Is the format sufficiently useful to include in LLVM?

There are two ways to think about this question. The first is a simple existence proof:

Since my original RFC, the gnu toolchain has added support for two additional targets, and others are under consideration. It has reasonably good traction there. Kernel developers are also adopting it via the gnu toolchain. A cursory glance at related mailing lists shows perhaps a dozen companies involved in its development in one way or another.

The contributors have varying levels of technical sophistication, but I’m sure most are aware of the various other unwind solutions in the pipeline. And they still have concluded this is a useful project to pursue. Regardless of what any one person thinks of it, enough companies and people have done enough work to prove that a broad group expects it will be a useful feature.

The second way of thinking about its usefulness is a more technical analysis. This is what some internal evaluations of the different options have concluded:

[@davidxl] Shadow stack support is limited to only a handful of targets. It is not ready to be turned on by default in production. Our performance experiments indicates non trivial performance regressions with it on.”

“[@rosedt] We have evaluated alternatives (see shadow stack reply). For kernel, there’s two use cases. One is the use case for profiling user space applications and the other is the use of using it as the unwinder in the kernel for live patching.

The hardware assisted use case can help with the former, but it is usually limited to a fix number of elements. If you have a stack deeper than that limit, sframes can still prove useful.

For the live kernel patching, hardware assistance is not helpful, as we need to unwind stacks of tasks that are currently off the run queue, and may not have executed for some time.

Regarding the syscall handling and BPP, the Deferred stacktrace infrastructure has been accepted upstream. This is needed for SFrames, as the SFrame tables exist in user space. Profiling happens via an interrupt or NMI. In that context, user space tables are unsafe to read. The deferred infrastructure is a way to delay the user space stack reading until the task goes back to user space. In that context, the user space tables are safe to read.

It may be that other unwind mechanisms turn out to cover some of these use-cases, but that is far from obvious, and sframes have traction today. There is no one-size-fits-all solution, but SFrame is useful to a wide variety of use cases. Further, it is much easier for users to fully evaluate sframes with them in upstream LLVM.

In the near future, a toolchain without sframe support will not be viable for advanced linux kernel development. That alone is a strong case for including it, regardless of any other issues it may or may not have.

Should I have handled upstreaming this differently?

Although there were only a few comments on my original RFC, they were all positive. I left it open for several weeks. No one asked for more analysis or wondered about its future. I would have been happy to do that. Perhaps this is it. In fact, I have gotten more “glad to see this going in” comments for this work than anything else I have contributed to LLVM.

As far as how that patch was submitted: I generally accept any reviewer github suggests, without regard to whether they are my colleagues or not. Most of these PRs have had non-colleague approvals.

I apologize for not including you in the last one. I do tend to add people involved in the project as they have a motivation to review and tend to be more responsive. But I do welcome all comments–whether or not I take them.

The PR was open for a good two weeks before you commented. And I did accept all but one of your suggestions. Quite frankly, the MC changes were routine relaxation issues, following well-known patterns. I did not expect it to be controversial, but I will be more careful in the future.

Let us know if you have further concerns.

3 Likes

Adding a more high level note: Whether we like it or not, compiler projects have to interoperate with lots of external factors that we have varying degrees of control over. There may be design decisions in the C++ standard, the ELF format or the x86 ISA that we don’t agree with, but in most cases we don’t really have a choice but supporting them anyway. In some cases we may be able to influence the design (C++ standard more than the x86 ISA…) but often we still have to support past versions anyway.

It’s good to engage with people to try and improve future iterations of the SFrame design. However, I don’t think this is something we can block LLVM support on, given that the technology is starting to see increased adoption in various places and LLVM has to interoperate with the wider ecosystem. For example, multiple people have already reported issues with lld on Ubuntu 25.10, which enabled sframe emission in GCC by default.

3 Likes

SFrame as a new endeavor in metadata-based stack unwinding is interesting and has gained traction.
However, several aspects make it premature to support the current version.

Format instability: There are significant format differences between v2 and the upcoming v3.
While format evolution past v3 is expected, the magnitude of these changes from v2 to v3 is expected to be large, causing toolchain support chrun and indicating the design hasn’t stabilized sufficiently for production toolchain support.

Ongoing discussions GNU Tools Cauldron SFrame talk notes about ld -r behavior, that .sframe consumers should support multi-element structure, and the relationship between .sframe and .sframe_idx represent fundamental design questions that affect both producers and consumers.
They fundamentally change how SFrame will be used and processed.

Linkers building the current monolithic .sframe index (the majority of the complexity) design will face substantial rewrites when v3 is released.
This isn’t merely about version number increments - these are structural changes that will require significant code modifications.
The ~1000 lines of linker code we’re discussing represents a substantial investment that may need to be largely rewritten.

Binutils’ SFrame v3 TODO page (Making sure you're not a bot!) makes clear the breaking nature of this transition:

The updated linker will not, however, continue to link SFrame V2 sections.
The linker will not automatically upgrade input SFrame V2 sections to output a SFrame V3 section.

While they might reconsider supporting mix-and-match v2/v3, that would impose very significant burden on linker implementations.
A lot of complexity could be avoided by waiting until v3 (or a corrected v2 addressing the object file format issues) stabilizes.

Note that if SFrame consumers support multi-element structures, linker-based index building becomes optional.
An alternative approach would be a post-link tool that scans the potentially multi-element .sframe section and constructs the index.

Recommendation: I propose deferring the linker optimize-sframe support until the format stabilizes. The assembler and binary utilities that have landed can remain, allowing the community to experiment with the format and gain implementation experience.
However, committing substantial linker infrastructure to a design that’s about to undergo breaking changes seems premature. We can revisit linker support once v3 is finalized or v2 is corrected to address the ELF compliance and structural issues.


.eh_frame has a number of problems that have led to significant linker complexity-understandable given it’s a technology invented more than 20 years ago.
However, new metadata formats should not use “.eh_frame does it that way” as justification for similar misdesign.
.debug_frame as a non-SHF_ALLOC section, is not scanned for relocations, causing no discarded section or --gc-sections issue.

That said, I recognize the value of reducing the burden that SFrame support would impose on linkers.
To that end, I’ve been working on streamlining .eh_frame support in LLD, which would also benefit future SFrame implementation. Recent changes include:

https://github.com/llvm/llvm-project/pull/160031 ELF: Split relocateAlloc to relocateAlloc and relocateEh. NFC
https://github.com/llvm/llvm-project/pull/161041 ELF: Store EhInputSection relocations to simplify code. NFC
https://github.com/llvm/llvm-project/pull/161091 ELF: Use preprocessed relocations for EhInputSection scanning
1 Like

IMO a lot of things you described are overstated (with frequent use of 'very signicantly, fundamentally etc).

First of all, v3 won’t be finalized until next year at the soonest. The changes will require adjustments but are not particularly big. Whatever the changes, the philosophy isn’t changing all that much.

Secondly, it is more important to have a functional and well tested v2 framework in place when v3 comes along (whenever that is). Adapting the implementation to v3 will be much less work than writing from scratch from the testing. It is also useful to recommend more valuable suggestions to v3 from practical use cases encountered in the field.

Finally, whatever the extra complexity and extra work of supporting v2, the vast majority of costs and complexity are borne by those who actually want it in place. There will be some extra lines in the linker, but they are well defined and easy to identify. Those who don’t want to work on them can mostly just ignore them.

So it need not cost anyone but the actual implementers much at all.

1 Like

I will push back on that one. More code and features in LLD means more care needed when refactoring or otherwise making LLD changes that permeate through the implementation. Those adding SFrame support may be the ones bearing the cost of writing the first implementation (though let’s not forget that that still requires reviewer time and investment), but they’re not the ones who have to bear the cost of future changes that other people want to make. Something that MaskRay has done a lot of over the years in LLD, for example.

5 Likes

I believe that delaying sframe support until v3 is finalized next year would be a big mistake.

There are already issues with interoperability as mentioned by nikic above. Waiting until v3 is finalized, and then finally makes its way into distros means perhaps another year of incompatibility and issues.

Anyone else have any thoughts?

I don’t have a strong opinion on that matter (and I don’t maintain the code, either). I personally feel we could be slightly more conservative w.r.t. new and arguably pre-mature features that have a clear potential to become maintenance liabilities.

I couldn’t find a clear motivation for sframes in the first place; the linked page merely mentions stack tracing, but no actual use case. I assume this is for profiling? Or something else?

IMHO, sframe is a pre-mature design and has the potential to be a huge missed opportunity to improve stack unwinding in general. I think a new CFI format should ultimately aim to replace eh_frame for most functions while providing the same benefit that sframe provides now. I.e., integrate callee-saved registers, personality functions, LSDA pointers. Personally, I’d also love to see exploration of compact unwinding formats for x86, which should help both with tracing performance and code size.

That said, if there’s real need (some data could help here), I think a viable option could be to integrate v2 now (with the attached costs) while making sure that v3 is mature and powerful enough so that it doesn’t need to be replaced in the same fashion anytime soon.

2 Likes

About the real need for SFrame support, Sterling has provided answers up-thread. I will repeat it here with addition information:

There’s two use cases for kernel. One is the use case for profiling user space applications and the other is the use of using it as the unwinder in the kernel for live patching.

Regarding the syscall handling and BPP, the Deferred stacktrace infrastructure has been accepted upstream. This is needed for SFrames, as the SFrame tables exist in user space. Profiling happens via an interrupt or NMI. In that context, user space tables are unsafe to read. The deferred infrastructure is a way to delay the user space stack reading until the task goes back to user space. In that context, the user space tables are safe to read.

Deferred stacktrace infrastructure has been accepted upstream.

Also important to note here that in the four months since the RFC an additional three target architectures have added or started support inside the gnu toolchain (s390, longarch, and risc-v).

To quote nikic@'s reply: “multiple people have already reported issues with lld on Ubuntu 25.10, which enabled sframe emission in GCC by default.”

Lacking SFrame support will be a major feature gap for kernel build with clang toolchains.

Another major use of the feature is for better performance: omit-fp can be enabled with low profiling (with stack trace) overhead.

David

As a legacy system, OpenVMS does have interesting rules which make us require frame-pointers and LSDAs so sframe in its current form isn’t interesting to me. Plus we have our own linker, our own image loader, and our own unwind library (it does utilize FreeBSDs libunwind() more-or-less).

However, I also have perspective from systems with dense and fast unwind information on our older VAX and Alpha platforms. Unwinding was fast and almost trivial (at the expense of a few limits like only one prologue and only one epilogue). That encouraged customers to write applications that had heavy unwind/stack-walking as part of their algorithms. When we ported to Itanium, that hit a brick wall with the ugly and complex unwind descriptors. We had to make extensive modifications to the unwind library to attempt to cache EH data, etc. It was never as fast as Alpha.

Now when porting to x86, we were faced with the EH/CFI model we have today along with the poor LLVM implementation (at the time) of prologue/epilogue EH data (we have platform rules that require different behavior for asynchronous events during prologue/epilogue). We also have platform specific pseudo-registers (essentially the entire Alpha 32 register set) that makes the CFI table have more columns.

There is the Apple-specific compact EH in LLVM today (and it looks much like the Alpha form we had back in the 1990s) and we even had a short discussion on the mailing list about wanting to enable it for non-Apple x86 targets. Our Calling Standard even discusses it and how we could support a mixture of both compact and traditional CFI. We received little response or support on the mailing list.

Now with the sframe discussion, I’m pleased to see the need for a more dense/faster EH model, but it feels like we are chasing ghosts or incomplete solutions especially for LLVM consumers that are out-of-tree and on niche platforms. We’ve made several out-of-tree changes to MC for OpenVMS’s requirements.

I want to thank MaskRay for the excellent writeup on his blog (and here on Discourse). That was good data to collect in a single location.

This might be a good topic to discuss at a roundtable in San Jose later this month.

I should note i have the same concerns about kcfi but in the opposite direction. It was implemented in clang/llvm already but gcc developers (myself included) noticed that the hashing used for kcfi does not work for many valid c code. And some of us want the hashing to change due to it being wrong and based on the mangled name seems to be the wrong design.

Now if sframe support is waited on, I am going to push back hard on getting kfci fixed too.

Thanks for the additional context. Are there live patching users with Clang/LLD-built Linux? If not, how is this related to kernel builds?

For context, I think it’s this thread: [llvm-dev] [RFC] Improving compact x86-64 compact unwind descriptors

(Conceptually I’d be in favor, but I haven’t assessed the proposal in detail w.r.t. shrink wrapping and multiple epilogues.)

+1!

As I can’t make it to the US dev meetings, if there’s discussion on this topic, I’d appreciate if someone could post a summary to Discourse.

yes of course. To name a few. Google and Meta’s production kernel is built with clang toolchains and live patching is needed.

Linker-based index building for SFrame cannot be optional. Stack tracers cannot take the performance hit of creating the index at run time. This directly conflicts with the usecase of SFrame: fast stack tracing.

A post-link tool will be deterrent for distro wide adoption as it necessitates changes to per-package build workflow.

Re: adding --sframe-index via a post-link step, there are at least two issues that make it an unacceptable solution. Both boil down to the fact that debug-info is not a good model for this problem.

The first is that sframes (and the index itself, in whatever form), needs to be loadable, and indeed, *loaded* at the point the stack trace is taken, possibly by the kernel itself. Adding a section like .gdb_index post-link is simply a matter of objcopy --add-section=… It doesn’t need to be loaded, or have any special handling by anything other than the debugger. Adding a loadable segment post-link is substantially more complicated.

Building it at runtime–even at startup–would impose costs high enough to make it unusable.

The second is that even though gdb_index is easy to add post-link, users thought it was sufficiently annoying that we added linker support in three linkers: gnu-ld, gnu-gold, and lld itself. These additions were completely uncontroversial. That’s a pretty big testament to the value of having the linker do it. It’s easy to add a linker flag, but much harder to add a post-link build step. Especially with things like build-ids, read-only file systems, and whatever else. Adding sframes would mean every package in every distro would need to add a post-link build step, and a distro would have a very hard time enabling it by default without hundreds of packages updating themselves.

I have only anecdata here, but I suspect somewhere around 90% of gdb_index users do it at the link step, rather than post-link. And that’s the best-case scenario.

I think you’re placing a very high premium on the cost of one thousand lines of code. Code is often temporary, and is rewritten frequently. It’s the interfaces that endure.

Maintenance burden is an important consideration, but how can we grow new LLD maintainers if the project has such a conservative stance that new contributors can’t add features to the linker? The way open source normally works is someone proposes a feature, stakes out some space in the repository, implements it behind a flag, iterates on it, refactors it, integrates it into the system, and as they iterate on that, they take on maintenance responsibilities. The bet on inclusivity may not pay off in all cases, and I’m sure we can point to our favorite poorly-thought-out and unmaintained feature, but I firmly believe that this is how you build sustainable open source projects.

“Open source succession planning” has been a hot topic recently (1, 2), and it’s never too early to start building depth in the maintainer pool. It comes at the cost of accepting increased project scope, so one has to weigh the tradeoffs, but I think in the case of LLD, there’s arguably a project health issue where the project is dependent on one contributor:

❯ git log --pretty=format:"%an" --since='2 year' ../lld/ELF | sort | uniq -c | sort -nr | head | cut -b1-4
 377
  19
  15
  14
  13
  12
   7
   6
   6
   6

I agree with this, but I don’t think we should let perfect be the enemy of the good. This format exists, we should support it, users will get value from it, and we should roll the experience from usage in the field into the next format version.

Not to cast too many aspersions, but sframes feel like something kernel developers came up with to solve a problem with very narrow scope, rather than accepting a 30% scope increase and building something that solves general problems. A recurring pattern talking to kernel developers is “step 1, let’s leave C++ concerns out of scope”, so later on down the road someone has to go back and come up with a new compact unwind info format that supports exception handling.

There’s this meme that “win32 is the only stable ABI on Linux”, but my hot take is that, as members of the Linux OSS developer tools community, this should actually liberate us to try out more new experimental features, like crel. ELF development has stagnated, and there’s a lot of room for improvement (think @maskray’s lightelf post), and the best way to make that happen is to ship the features we have today (sframes), and then deprecate and replace with ones that are incrementally better. Critical design feedback is valuable, but if we can’t build and ship new features until their perfect, we’re going to be stuck with the same inefficient file formats we have for decades on end.

1 Like