About phoneme-level SSL feature

[Secondtonumb]

Hi, thank you very much for your excellent work — it has been extremely helpful.

I applied the Kaldi GOP method to extract phoneme-level timestamps, and based on these timestamps, I further processed the SSL features as follows:
1. Use GOP-generated timestamps to segment frame-level SSL features.
2. Aggregate frame-level features within each phoneme segment into a chunk.
3. Compute the mean of each chunk as the phoneme-level SSL feature.

def extract_utterance(self, utt_id: str, wav_path: str) -> np.ndarray:
        """
        提取单个 utterance 的音素级特征
        
        Args:
            utt_id: utterance ID
            wav_path: 音频文件路径
        
        Returns:
            (num_phonemes, target_phn_frames, feature_dim) 的特征数组
        """
        # 提取帧级特征
        frame_features = self._load_and_extract_features(wav_path)
        
        # 获取该 utterance 的音素信息
        if utt_id not in self.ctm_data:
            print(f"Warning: {utt_id} not found in CTM file")
            return np.array([], dtype=np.float32)
        
        phones_info = self.ctm_data[utt_id]
        phn_features_list = []
        
        # 对每个音素进行池化
        for phn_info in phones_info:
            start_frame = self._time_to_frame_idx(phn_info['start_time'])
            end_frame = self._time_to_frame_idx(phn_info['end_time'])
            
            # 提取该音素的帧
            start_frame = max(0, start_frame)
            end_frame = min(end_frame, frame_features.shape[0])
            
            if start_frame < end_frame:
                phn_frames = frame_features[start_frame:end_frame]
            else:
                phn_frames = np.array([], dtype=np.float32).reshape(0, self.feature_dim)
            
            # 池化
            pooled = self._pool_frames(phn_frames) # mean here
            phn_features_list.append(pooled)
        
        # 堆叠所有音素特征
        if phn_features_list:
            phn_features = np.stack(phn_features_list, axis=0)  # (num_phonemes, feature_dim)
        else:
            phn_features = np.array([], dtype=np.float32).reshape(0, self.feature_dim)
        
        # Padding
        padded_features = self._pad_features(phn_features)
        return padded_features

However, after doing this, I found that the extracted phoneme-level features differ significantly from those you released and the performance gap is also quite noticeable.

So I would like to ask:
1. Is this chunk-mean aggregation strategy consistent with your actual implementation?
2. Did you perform any additional preprocessing steps on the SSL features before or after phoneme-level aggregation?
3. Do the neighbouring silent frames also count?

This would be extremely helpful if you could help me check these issues.

Here I attached:
• My generated CTM file on test set.
• The corresponding phoneme-level WavLM features extracted using the method above and your release on utt 000030012

feats_and_ctm.zip

[Secondtonumb]

Update:

I reimplemented the experiment using my own phoneme-level features and obtained results that are quite similar to those reported in the paper, except for the completeness metric.

However, I noticed that the feature heatmaps look quite different between the released features and mine. In particular, the released features appear more clearly segmented, whereas mine are comparatively smoother.

[a2d8a4v]

Hi, @Secondtonumb
I appreciate you interested in our work.

The process to generate aggregated SSL features is mostly correct. (Note: I think it's a typo, GOP not GOPT)
I've seen the files you provided, the shapes in original and yours edition matches, but values vary a lot.
I think it's something different inside the _load_and_extract_features function, and the way to computing the start_frame and end_frame
Can you also explain how you implement this part, including the padding method, the layer you used, etc.

[Secondtonumb]

I used the last layer of SSL features.

    def _load_and_extract_features(self, wav_path: str) -> np.ndarray:
        """
        加载音频并使用 SSL 编码器提取帧级特征
        
        Args:
            wav_path: 音频文件路径
        
        Returns:
            (num_frames, feature_dim) 的特征数组
        """
        try:
            # 加载音频
            waveform, sample_rate = torchaudio.load(wav_path)
            
            # 重采样到 16kHz（如果需要）
            if sample_rate != 16000:
                resampler = torchaudio.transforms.Resample(sample_rate, 16000)
                waveform = resampler(waveform)
            
            # 确保是单声道
            if waveform.shape[0] > 1:
                waveform = waveform.mean(dim=0, keepdim=True)
            
            # 移到 GPU
            waveform = waveform.to(self.device)
            
            # 提取特征
            with torch.no_grad():
                # 计算 wav_lens（相对长度，0-1之间）
                wav_lens = torch.tensor([1.0], device=self.device)
                
                # # SSL encoder 的调用方式：model(wavs, wav_lens)
                # # 返回值通常是 (batch_size, num_frames, feature_dim)
                features = self.ssl_encoder(waveform, wav_lens)
            
            # 如果返回的是多层的，取最后一层
            if isinstance(features, (list, tuple)):
                features = features[-1] if isinstance(features, (list, tuple)) else features
            
            # 转换为 numpy
            features = features.cpu().numpy()  # (batch_size, num_frames, feature_dim)
            
            # 移除批次维度
            if len(features.shape) == 3:
                features = features[0]  # 第一个样本
            elif len(features.shape) == 2:
                pass  # 已经是 (num_frames, feature_dim)
            
            return features.astype(np.float32)
        
        except Exception as e:
            print(f"Error loading {wav_path}: {e}")
            import traceback
            traceback.print_exc()
            raise

self.ssl_encoder is a wrapper for different types of SSL models, following the SpeechBrain implementation (link).

I also tried the original HuggingFace SSL implementation, which produced very similar results (cosine similarity ≈ 0.99). Therefore, I do not believe the discrepancy is caused by the wrapper.

[a2d8a4v]

Pardon,
I think I may mislead you how to extract the features.

The audio waveform is first chunked with phoneme timestamps, and then conveyed to SSL feature extractor.
I use huggingface as example, so you can reproduce with SpeechBrain.

input_values = processor(speech[start_time:end_time], padding='max_length', max_length=400, return_tensors="pt", sampling_rate=16000).input_values
It's notably that the input is padded to 400 if input less than 400 samples

I just recalled the method you're trying now. The efficacy of getting full length of SSL feature and chunking would have lower performance than the one I said above (I tried that before). I think it is due to the smoothing problem when long form audio is inside the SSL models.

[Secondtonumb]

It's notably that the input is padded to 400 if input less than 400 samples。

I see—this is a quite important detail. This suggests that many consonants may be affected by this padding behavior.

The efficacy of getting full length of SSL feature and chunking would have lower performance than the one I said above (I tried that before).

That’s very interesting. I’ll try to implement your method accordingly.

[Secondtonumb]

You are right. After applying the clip-first strategy, the extracted features become much more similar to yours, and the completeness score increased as well.

[a2d8a4v]

Do you think it would be an interesting research issue on this topic?
With using clip-first strategy may help SSL models to focus on each specific phoneme at their corresponding timestamps, it's unsure that whether it is really good to forcely make audio representation independently for real-world practical use (Although their relationships are learned while inside GOPT or HierCB models with the attention mechanism. and the metrics show promoting results)
I was wondering if a speaker tending to make wrong on some specific pronunciations, those would have strong correlation.
Some paper works done by @bicheng1225 also use graph modeling to do this trick (e.g. using error pattern matrix), making sense on such cases.
I think it would be interesting if we can find out the differences in high-score and low-score pronunciations in their representatoin, and in different age, gender groups, etc.

[Secondtonumb]

Do you think it would be an interesting research issue on this topic?
forcely make audio representation independently for real-world practical use

I believe segmentation is a key bottleneck for practical usage (e.g., spontaneous speech assessment). As we discussed last time, our team is also exploring ways to make APA applicable to free-spoken language, where HiPPO seems to be a promising and pioneering attempt :)

Some paper works done by @bicheng1225 also use graph modeling to do this trick (e.g. using error pattern matrix).

I agree that incorporating prior knowledge and error graph (like Peppenet or MDDGCN ) can be effective, particularly at the corpus-level. As those speakers in the same corpus can exhibit correlated mispronunciation patterns. However, I think it's an open question whether such prior-knowledge-injecting approaches generalise well to broader L2 populations with diverse backgrounds.

Nevertheless, more fine-grained investigations along this direction would be nice :)