create_llvm_prof is not working when perf version is 6.12

[ptr1337]

Hi together,

Recently on cachyos we have updated the linux-tools perf to the 6.12 release.
When trying to convert the profile with perf 6.12 I get following:

 create_llvm_prof --binary=/usr/src/debug/linux-cachyos/vmlinux --profile=/home/ptr1337/bench/withoutprime.data --format=extbinary --out=cachyos-benchmarker.afdo
[ERROR:/home/runner/work/autofdo/autofdo/third_party/perf_data_converter/src/quipper/perf_reader.cc:161] Event size 0 of event UNKNOWN_EVENT_0 is less than header size 8
[ERROR:/home/runner/work/autofdo/autofdo/third_party/perf_data_converter/src/quipper/perf_reader.cc:1335] Error reading build ID header.
WARNING: Logging before InitGoogleLogging() is written to STDERR
E20241205 17:53:35.636153 254754 profile_creator.cc:209] Error reading profile from /home/ptr1337/bench/withoutprime.data

When using perf 6.10 it does work correctly:

create_llvm_prof --binary=/usr/src/debug/linux-cachyos/vmlinux --profile=/home/ptr1337/bench/withoutprime.data --format=extbinary --out=cachyos-benchmarker.afdo
[WARNING:/home/runner/work/autofdo/autofdo/third_party/perf_data_converter/src/quipper/perf_reader.cc:1322] Skipping 268 bytes of metadata: HEADER_CPU_TOPOLOGY
[WARNING:/home/runner/work/autofdo/autofdo/third_party/perf_data_converter/src/quipper/perf_reader.cc:1069] Skipping unsupported event PERF_RECORD_ID_INDEX
[WARNING:/home/runner/work/autofdo/autofdo/third_party/perf_data_converter/src/quipper/perf_reader.cc:1069] Skipping unsupported event PERF_RECORD_EVENT_UPDATE
[WARNING:/home/runner/work/autofdo/autofdo/third_party/perf_data_converter/src/quipper/perf_reader.cc:1069] Skipping unsupported event PERF_RECORD_CPU_MAP
[WARNING:/home/runner/work/autofdo/autofdo/third_party/perf_data_converter/src/quipper/perf_reader.cc:1069] Skipping unsupported event UNKNOWN_EVENT_17
[WARNING:/home/runner/work/autofdo/autofdo/third_party/perf_data_converter/src/quipper/perf_reader.cc:1069] Skipping unsupported event UNKNOWN_EVENT_82
[INFO:/home/runner/work/autofdo/autofdo/third_party/perf_data_converter/src/quipper/perf_reader.cc:1060] Number of events stored: 4931669

Reproducer:

1. Go into cachyos or archlinux enviroment
2. Compile kernel with autofdo support and an unstripped vmlinux
3. Boot into the kernel and profile a workload with perf record --pfm-events RETIRED_TAKEN_BRANCH_INSTRUCTIONS:k -a -N -b -c 500009 -o cachyos-benchmarker.data -- cachyos-benchmarker ~/bench
4. Try to convert the profile: create_llvm_prof --binary=/usr/src/debug/linux-cachyos/vmlinux --profile=/home/ptr1337/bench/withoutprime.data --format=extbinary --out=cachyos-benchmarker.afdo

If you need further logs, please let me know

[ptr1337]

Little update:
perf 6.11 is working. So its a regression/change in the 6.12 release.

If you could help me debugging this down, I would like to report ustream

[shenhanc78]

Thanks for reporting. We are looking into it and will report back shortly.

[shenhanc78]

Hi Peter, I am trying to reproduce.

I worked on a archlinux, updated to newest with kernel 6.12.4 and perf 6.12-1. Seems the system is missing libpfm support, and I cannot find libpfm via pacman. Then I saw you patch to add support and I tried to makepkg on your change but failed with libpfm not found error on my archlinux. Could you briefly describe how to add libpfm support for arch, thanks.

-Han

[ptr1337]

Hi Peter, I am trying to reproduce.

I worked on a archlinux, updated to newest with kernel 6.12.4 and perf 6.12-1. Seems the system is missing libpfm support, and I cannot find libpfm via pacman. Then I saw you patch to add support and I tried to makepkg on your change but failed with libpfm not found error on my archlinux. Could you briefly describe how to add libpfm support for arch, thanks.

-Han

Hi,

I have pushed a perf version with working pfm support now into the extra-testing repository.
Please fetch the libpfm and perf package from there. I have verified that pfm events are working.

You can find them here (Click the download button and then install with sudo pacman -U):
https://archlinux.org/packages/extra-testing/x86_64/libpfm/
https://archlinux.org/packages/extra-testing/x86_64/perf/

[shenhanc78]

Hi Peter, thanks for preparing the packages. I am now able to reproduce.

The perf data parser, that this tool depends on, failed to parse "struct perf_event_header" (all fields are filled with zero), this may be caused by (unlikely) the new perf-6.12 tool's failing to correctly write the header into perf.data, or (probably) the parser has a bug or is incapable of parsing the new perf-6.12. format (I don't know what has changed in 6.12). I'll keep looking.

[shenhanc78]

Filed an internal bug regarding perf data parser, will report back.

[shenhanc78]

We found the error happens during reading build-id, and we then found this.

After apply the patch on top of linux-tools-6.12-7 , the "Error reading build ID header." is gone.

diff --git a/tools/perf/util/build-id.c b/tools/perf/util/build-id.c                                                                                          
index 8982f68e7230..4baf925beb1f 100644                                                                                                                       
--- a/tools/perf/util/build-id.c                                                                                                                              
+++ b/tools/perf/util/build-id.c                                                                                                                              
@@ -277,7 +277,8 @@ static int write_buildid(const char *name, size_t name_len, struct build_id *bid                                                          
        struct perf_record_header_build_id b;                                                                                                                 
        size_t len;                                                                                                                                           
                                                                                                                                                              
-       len = sizeof(b) + name_len + 1;                                                                                                                       
+       /* len = sizeof(b) + name_len + 1; */                                                                                                                 
+       len = name_len + 1;                                                                                                                                   
        len = PERF_ALIGN(len, sizeof(u64));                                                                                                                   
                                                                                                                                                              
        memset(&b, 0, sizeof(b));                                                                                                                             
@@ -286,7 +287,8 @@ static int write_buildid(const char *name, size_t name_len, struct build_id *bid                                                          
        misc |= PERF_RECORD_MISC_BUILD_ID_SIZE;                                                                                                               
        b.pid = pid;                                                                                                                                          
        b.header.misc = misc;                                                                                                                                 
-       b.header.size = len;                                                                                                                                  
+       /* b.header.size = len; */                                                                                                                            
+       b.header.size = sizeof(b) + len;                                                                                                                      
                                                                                                                                                              
        err = do_write(fd, &b, sizeof(b));                                                                                                                    
        if (err < 0)

[nickdesaulniers]

cc @captain5050, perhaps a bug in https://github.com/googleprodkernel/linux-perf/commit/ae39ba16554eb4a2d97f752847e93aa428438912.

[captain5050]

I believe this is fixed by: https://lore.kernel.org/r/Z0aRFFW9xMh3mqKB@google.com

[nickdesaulniers]

Does that need to get cherry-picked back into 6.12?

[ptr1337]

Thanks! The fix appears to work from Namhyung Kim

[captain5050]

The fixes tag means it should be pulled automatically.