Unify `PDMat` and `PDSparseMat` + move SparseArrays support to an extension by devmotion · Pull Request #188 · JuliaStats/PDMats.jl

devmotion · 2023-10-04T07:39:05Z

This PR unifies PDMat and PDSparseMat and makes PDSparseMat an (unexported) alias of PDMat. This makes it also possible to move support for SparseArrays to an extension, without any workarounds such as non-functional types in the main package (see #174).

Initially, PDMat and PDSparseMat only supported Matrix{Float64} and SparseMatrixCSC{Float64} matrices. This restriction was lifted in #37 but the separate types with support for Cholesky and CHOLMOD.Factors were kept.

Due to the refactoring of SuiteSparse and SparseArrays (it was integrated in SparseArrays in Julia 1.9), on Julia >= 1.9 only the SparseArrays dependency is needed and SuiteSparse only loads functionality from SparseArrays (https://github.com/JuliaSparse/SuiteSparse.jl/blob/e8285dd13a6d5b5cf52d8124793fc4d622d07554/src/SuiteSparse.jl). It seemed inconvenient to demand that users on Julia >= 1.9 load both SparseArrays and SuiteSparse for the sparse extension, and hence I decided to let the extension only depend on SparseArrays. IIUC, however, this means that we can't support Julia < 1.9 anymore since there SparseArrays does not contain the required CHOLMOD functionality. @dkarrasch, is this correct?

Since this means we drop support for the current LTS version, my suggestion would be to backport bugfixes to older Julia versions (at least >= 1.6) until another Julia version >= 1.9 becomes the new LTS version.

Edit: The PR keeps support for Julia 1.0-, I found a workaround: #188 (comment)

Fixes #174.

codecov-commenter · 2023-10-04T07:43:01Z

Codecov Report

❌ Patch coverage is 96.87500% with 5 lines in your changes missing coverage. Please review.
✅ Project coverage is 94.13%. Comparing base (473f638) to head (bc89bea).

Files with missing lines	Patch %	Lines
ext/PDMatsSparseArraysExt/pdsparsemat.jl	97.75%	2 Missing ⚠️
ext/StatsBaseExt.jl	83.33%	2 Missing ⚠️
src/pdmat.jl	97.05%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #188      +/-   ##
==========================================
+ Coverage   93.37%   94.13%   +0.75%     
==========================================
  Files          11       13       +2     
  Lines         800      767      -33     
==========================================
- Hits          747      722      -25     
+ Misses         53       45       -8

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

dkarrasch

Thank you so much for taking up the challenge and moving this forward! I think this is a great service to the ecosystem. Perhaps we can make this work even for older Julia versions by using SuiteSparse or SparseArrays.SuiteSparse in the extension? On v1.9+, this should be a no-op, but on older versions (without Pkg extensions) this will just get loaded by including the extension code files. I'm not sure though if that would require "version-dependent" Project.toml files or other hassle.

devmotion · 2023-10-04T09:59:27Z

Perhaps we can make this work even for older Julia versions by using SuiteSparse or SparseArrays.SuiteSparse in the extension? On v1.9+, this should be a no-op, but on older versions (without Pkg extensions) this will just get loaded by including the extension code files. I'm not sure though if that would require "version-dependent" Project.toml files or other hassle.

AFAICT on older Julia versions we would have to keep both SparseArrays and SuiteSparse as dependencies. However, if only Julia >= 1.9 the extension only depends on SparseArrays, then SuiteSparse would still be a regular dependency and since it depends on SparseArrays the extension would be loaded unconditionally. Alternatively, the extension could depend on both SparseArrays and SuiteSparse (or only SuiteSparse) - but then users would have to load both SparseArrays and SuiteSparse (or only SuiteSparse). Both alternatives seem suboptimal.

However, I just thought of another option that I will try later: Making SparseArrays and SuiteSparse both a weak dependency but letting the extension only depend on SparseArrays.

devmotion · 2023-10-04T11:04:10Z

Seems to work 🙂 On Julia 1.0 and 1.6, PDMats depends on SparseArrays and SuiteSparse (https://github.com/JuliaStats/PDMats.jl/actions/runs/6404900221/job/17386347115?pr=188#step:5:36, https://github.com/JuliaStats/PDMats.jl/actions/runs/6404900221/job/17386347711?pr=188#step:5:27) and on Julia 1.9 and nightly neither SparseArrays nor SuiteSparse are dependencies (https://github.com/JuliaStats/PDMats.jl/actions/runs/6404900221/job/17386348265?pr=188#step:5:24, https://github.com/JuliaStats/PDMats.jl/actions/runs/6404900221/job/17386348761?pr=188#step:5:24) and the tests pass successfully with installing and loading only SparseArrays.

devmotion · 2023-10-17T07:00:06Z

I'll update the PR once the other PRs (non-breaking bugfixes and possibly a new feature) are merged.

timholy

Nice work!

One other thing we might want to look at, quite a few of the methods starting with ### (un)whitening rely on the Choleksy factorization. If there isn't time to review/generalize those algorithms, perhaps it might make sense to define a PDMatCholesky{T,S<:AbstractMatrix{T}} = PDMat{T,S,<:Cholesky} and then restrict the argument of those methods to be a PDMatCholesky? That way we're at least not promising things we can't deliver.

timholy · 2023-11-21T09:58:52Z

 end

-function PDMat(mat::AbstractMatrix,chol::Cholesky{T,S}) where {T,S}
+function PDMat(mat::AbstractMatrix{T}, chol::Cholesky) where {T<:Real}


Suggested change

function PDMat(mat::AbstractMatrix{T}, chol::Cholesky) where {T<:Real}

function PDMat(mat::AbstractMatrix{T}, fmat::Factorization) where {T<:Real}

Maybe we should wait with generalizing the constructor until we actually support and test other factorizations.

Co-authored-by: Tim Holy <tim.holy@gmail.com>

timholy

Needs the conflicts resolved, but otherwise LGTM.

timholy · 2024-01-12T13:00:37Z

Bump

devmotion · 2024-01-13T22:21:28Z

The PR is the result of some discussions I had with @andreasnoack about PDSparseMat and the SparseArrays dependency of PDMats, so I'd prefer to wait for his review.

timholy · 2024-01-13T22:54:12Z

Right, I wasn't bumping you 🙂

timholy · 2024-08-07T20:32:06Z

+struct PDMat{T<:Real,S<:AbstractMatrix{T},F<:Factorization} <: AbstractPDMat{T}
    mat::S                      # input matrix
-    chol::Cholesky{T,S}         # Cholesky factorization of mat
+    fact::F                     # factorization of mat


One needs an inner constructor that checks isposdef(fact) before calling new. Otherwise I could call

E = eigen(A) PDMat{eltype(A),typeof(A),typeof(E)}(A, E)

even if A is not posdef.

Currently none of the other constructors enforces positive semi-definite matrices: #22

So ideally this should be addressed more generally, to ensure that such checks are performed by all AbstractPDMat types. But I wonder if it should be left for a separate PR?

Sure. But it's not a hard change, we should just do it once other dust has settled. Given that the PDMat wrapper promises positive-definiteness, it is kinda important to enforce this, especially if we widen the type definition so that it can no longer be guaranteed from the input types alone.

timholy · 2024-08-07T20:39:05Z

+            @test M isa PDMat{Int,<:SymTridiagonal,<:Cholesky}
+            @test M == S
+        end
+    end


Probably want to add tests for what happens if one tries to construct with factorizations besides cholesky.

dkarrasch · 2025-02-21T09:49:22Z

Now that the lower compat has been raised to v1.10, this would be worthwhile to revisit and finish. I don't have current loading/latency times, but at the time this would have helped avoiding to load SparseArrays.jl unconditionally and therefore help all the dependent packages to keep loading times low(er).

Resolve conflicts from master, preserving all master improvements while keeping the PR's unification of `PDMat`/`PDSparseMat` and the SparseArrays package extension. Key resolutions: - `PDMat` keeps the generalized 3-parameter struct (`mat::S`, `fact::F`); master's coercing constructors and `convert` methods are re-expressed against `fact`/`cholesky(...)` (with an added 3-parameter `convert`). - Dense Cholesky operations stay on the general `PDMat`; `cholesky` is the only method restricted to the `PDMatCholesky` alias. The sparse extension overrides the CHOLMOD-specific methods by dispatch. - Master's sparse `invwhiten`/`invunwhiten` methods and modernized `quad`/`quad!` are folded into the extension's `pdsparsemat.jl`; generic sparse operations now flow through the unified core `PDMat` methods. - `congruence.jl` and `StatsBaseExt.jl` are narrowed to `PDMatCholesky` and switched from `.chol` to the `cholesky(...)` accessor (avoids catching sparse matrices and resolves dispatch ambiguities). - Preserved master features: coercing constructors/convert (#212, #210), `invwhiten`/`invunwhiten` (#221), `ldiv!` (#228), the `kron` fix (#218), ForwardDiff-safe `whiten!`/`unwhiten!` (#217), `congruence.jl`, the StatsBase extension, and the Runic formatting (#220). - Dropped pre-1.9 / Julia-1.0 compat shims (compat is now Julia >= 1.10); dropped the now-unused SuiteSparse dependency. - Fixed a latent dead `quad`/`invquad` specialization (`<:Vector` -> `Matrix{T}`). - Version bumped to 0.12.0 (breaking: `PDMat` struct parameters changed). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Unify PDMat and PDSparseMat

5ab9584

dkarrasch reviewed Oct 4, 2023

View reviewed changes

Comment thread test/testutils.jl Outdated

Comment thread ext/PDMatsSparseArraysExt/PDMatsSparseArraysExt.jl

devmotion added 3 commits October 4, 2023 12:03

Replace tab with whitespace

e4ed4ed

Try to add back support for Julia < 1.9

91f6d14

Fix support on older Julia versions

e817ceb

devmotion added 2 commits October 24, 2023 00:34

Merge branch 'master' into dw/unify_pdmat_pdsparsemat

dee9225

Fix test errors on Julia 1.0

dd947d6

devmotion mentioned this pull request Nov 9, 2023

X_A_Xt symmetric output breaks addition and subraction typing with static PDMats #195

Closed

Update README.md

2dc005d

This was referenced Nov 20, 2023

Other factorizations: parametrization or new names? #198

Open

Handle SparseArrays via Pkg extention? JuliaStats/Distributions.jl#1802

Closed

timholy reviewed Nov 21, 2023

View reviewed changes

devmotion and others added 7 commits November 21, 2023 15:56

Improve type parameter

deb575d

Co-authored-by: Tim Holy <tim.holy@gmail.com>

Rename field chol to fact

c27aab2

Merge branch 'master' into dw/unify_pdmat_pdsparsemat

578e459

Fix type parameter

6ce4b77

Fix variable conflicts with type parameters

bbedcb2

Clean up switch to fact

dca9273

Specialize sqrt for PDSparseMat

33a80f3

timholy approved these changes Nov 27, 2023

View reviewed changes

Merge branch 'master' into dw/unify_pdmat_pdsparsemat

60df321

devmotion requested a review from andreasnoack November 27, 2023 13:35

Merge branch 'master' into dw/unify_pdmat_pdsparsemat

6678dd3

devmotion mentioned this pull request Jul 30, 2024

Separate parent matrix type from Cholesky parameter #207

Closed

devmotion added 2 commits August 5, 2024 12:17

Add test from #207

a0acc49

Run SymTridiagonal tests only on Julia >= 1.8

8c160df

timholy reviewed Aug 7, 2024

View reviewed changes

andreasnoack removed their request for review November 5, 2024 14:25

devmotion mentioned this pull request Dec 10, 2024

More conversion in PDMat #212

Merged

timholy mentioned this pull request Dec 23, 2025

Handle Symmetric/Hermitian in PDSparseMat #230

Open

This was referenced Jun 24, 2026

Fix a few bugs #236

Merged

PDMat cannot pair a dense mat with a structured/lazy Cholesky factor (shared type parameter S) #237

Closed

	function PDMat(mat::AbstractMatrix{T}, chol::Cholesky) where {T<:Real}
	function PDMat(mat::AbstractMatrix{T}, fmat::Factorization) where {T<:Real}

Uh oh!

Conversation

devmotion commented Oct 4, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov-commenter commented Oct 4, 2023 • edited by codecov Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

dkarrasch left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

devmotion commented Oct 4, 2023

Uh oh!

devmotion commented Oct 4, 2023

Uh oh!

devmotion commented Oct 17, 2023

Uh oh!

timholy left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

timholy Nov 21, 2023

Choose a reason for hiding this comment

Uh oh!

devmotion Nov 22, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

timholy left a comment

Choose a reason for hiding this comment

Uh oh!

timholy commented Jan 12, 2024

Uh oh!

devmotion commented Jan 13, 2024

Uh oh!

timholy commented Jan 13, 2024

Uh oh!

timholy Aug 7, 2024

Choose a reason for hiding this comment

Uh oh!

devmotion Aug 7, 2024

Choose a reason for hiding this comment

Uh oh!

timholy Aug 8, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

timholy Aug 7, 2024

Choose a reason for hiding this comment

Uh oh!

dkarrasch commented Feb 21, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

devmotion commented Oct 4, 2023 •

edited

Loading

codecov-commenter commented Oct 4, 2023 •

edited by codecov Bot

Loading

timholy Aug 8, 2024 •

edited

Loading