[Merged by Bors] - Add TestUtils submodule by torfjelde · Pull Request #313 · TuringLang/DynamicPPL.jl

torfjelde · 2021-08-23T09:48:54Z

This PR adds a DynamicPPL.TestUtils submodule which is meant to include functionality to make it easy to test new samplers, new implementations of AbstractVarInfo, etc.

As of right now, this is mainly just a collection of models with equivalent marginal posteriors using the different features of DPPL, e.g. some are using .~, some are using @submodel, etc.

Eventually this should be expanded to be of more use, but more immediately this will be useful to test functionality in open PRs, e.g. #269, #309, #295, #292.

These models are also already used in Turing.jl's test-suite (https://github.com/TuringLang/Turing.jl/blob/9f52d75c25390b68115624b2e6cf464275a88137/test/test_utils/models.jl#L55-L56), so this PR would avoid the code-duplication + make it easier to keep things up-to-date.

devmotion · 2021-08-23T09:59:47Z

+
+# A collection of models for which the mean-of-means for the posterior should
+# be same.
+@model function demo1(x=10 * ones(2), ::Type{TV}=Vector{Float64}) where {TV}


Maybe we could use a bit more descriptive names?

Thought crossed my mind, but was worried they'd be too long. Buuuut we're not going to be using the constructors of these models very often, so the cost of making them overly verbose is approx. 0 👍

I'll do that!

phipsgabler

Aren't the implicitly constructed named tuples ((; m, x)) too new to be backwards compatible with 1.3?

Co-authored-by: Philipp Gabler <phipsgabler@users.noreply.github.com>

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

torfjelde · 2021-08-23T12:32:04Z

Is this ready to go, or is there something else that needs to be addressed?

phipsgabler · 2021-08-23T12:51:39Z

+    meanfunction,
+    sampler::AbstractMCMC.AbstractSampler,
+    args...;
+    target=8.0,


https://xkcd.com/221/?

Haha, not entirely certain what you're saying 😅 It's "random" in the sense that it just happened to be the number that you get from the default values of the models, but it's deliberate in the sense that it's the actual true mean of the posterior of the models with the default values:)

So it's really the expected result, but only if you pass meanfunction = mean? I suspected it's either that or a random you put in for testing.

So if IIUC, maybe that is better?

function test_sampler_demo_models( meanfunction, sampler::AbstractMCMC.AbstractSampler, args...; target, atol=1e-1, rtol=1e-3, kwargs..., ) ... end test_sampler_demo_models(::typeof(mean), args...; kwargs...,) = test_sampler_demo_models(mean, args...; target=8.0, kwargs...)

It's not valid just for mean though. This is because different samplers can have completely different return-values from sample, and so we want to allow different mean functions, while still wanting the target to be 8.0.

phipsgabler · 2021-08-23T12:58:16Z

Apart from the magic 8.0, looks good to me.

devmotion

Some additional suggestions 🙂

Co-authored-by: David Widmann <devmotion@users.noreply.github.com>

torfjelde · 2021-09-08T22:44:22Z

bors r+

This PR adds a `DynamicPPL.TestUtils` submodule which is meant to include functionality to make it easy to test new samplers, new implementations of `AbstractVarInfo`, etc. As of right now, this is mainly just a collection of models with equivalent marginal posteriors using the different features of DPPL, e.g. some are using `.~`, some are using `@submodel`, etc. Eventually this should be expanded to be of more use, but more immediately this will be useful to test functionality in open PRs, e.g. #269, #309, #295, #292. These models are also already used in Turing.jl's test-suite (https://github.com/TuringLang/Turing.jl/blob/9f52d75c25390b68115624b2e6cf464275a88137/test/test_utils/models.jl#L55-L56), so this PR would avoid the code-duplication + make it easier to keep things up-to-date.

bors · 2021-09-08T22:58:30Z

Build failed:

test (1, ubuntu-latest, x64, 2)

torfjelde · 2021-09-09T15:11:46Z

bors r+

This PR adds a `DynamicPPL.TestUtils` submodule which is meant to include functionality to make it easy to test new samplers, new implementations of `AbstractVarInfo`, etc. As of right now, this is mainly just a collection of models with equivalent marginal posteriors using the different features of DPPL, e.g. some are using `.~`, some are using `@submodel`, etc. Eventually this should be expanded to be of more use, but more immediately this will be useful to test functionality in open PRs, e.g. #269, #309, #295, #292. These models are also already used in Turing.jl's test-suite (https://github.com/TuringLang/Turing.jl/blob/9f52d75c25390b68115624b2e6cf464275a88137/test/test_utils/models.jl#L55-L56), so this PR would avoid the code-duplication + make it easier to keep things up-to-date.

bors · 2021-09-09T15:26:28Z

Build failed:

test (1.3, ubuntu-latest, x64, 1)

torfjelde · 2021-09-09T15:34:02Z

bors r+

This PR adds a `DynamicPPL.TestUtils` submodule which is meant to include functionality to make it easy to test new samplers, new implementations of `AbstractVarInfo`, etc. As of right now, this is mainly just a collection of models with equivalent marginal posteriors using the different features of DPPL, e.g. some are using `.~`, some are using `@submodel`, etc. Eventually this should be expanded to be of more use, but more immediately this will be useful to test functionality in open PRs, e.g. #269, #309, #295, #292. These models are also already used in Turing.jl's test-suite (https://github.com/TuringLang/Turing.jl/blob/9f52d75c25390b68115624b2e6cf464275a88137/test/test_utils/models.jl#L55-L56), so this PR would avoid the code-duplication + make it easier to keep things up-to-date.

bors · 2021-09-09T16:28:28Z

Pull request successfully merged into master.

Build succeeded:

torfjelde added 10 commits June 10, 2021 13:23

initial work on adding test utils

a5280ea

include test_utils

dd7d774

added testing for continuous samplers

caffa78

Merge branch 'master' into tor/test-utils

c8d6b1f

allow specification of target mean in test_sampler_continuous

7b58174

add return-values to the test models which can be useful

a1a9752

Merge branch 'master' into tor/test-utils

eb92fd1

Merge branch 'master' into tor/test-utils

fc815d0

added TestUtils submodule

b32eb59

added some docstrings to TestUtils

b36c842

devmotion reviewed Aug 23, 2021

View reviewed changes

phipsgabler reviewed Aug 23, 2021

View reviewed changes

Comment thread src/test_utils.jl Outdated

Comment thread src/test_utils.jl Outdated

torfjelde commented Aug 23, 2021

View reviewed changes

Comment thread src/test_utils.jl

torfjelde and others added 3 commits August 23, 2021 11:05

fix 1.3 compatibility

e0adc2b

test model names are now more informative

3b71318

Apply suggestions from code review

0d3ff26

Co-authored-by: Philipp Gabler <phipsgabler@users.noreply.github.com>

github-actions Bot reviewed Aug 23, 2021

View reviewed changes

Comment thread src/test_utils.jl Outdated

Comment thread src/test_utils.jl Outdated

Comment thread src/test_utils.jl Outdated

Comment thread src/test_utils.jl Outdated

Comment thread src/test_utils.jl Outdated

Apply suggestions from code review

334e8da

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

phipsgabler reviewed Aug 23, 2021

View reviewed changes

devmotion reviewed Aug 26, 2021

View reviewed changes

torfjelde and others added 2 commits August 26, 2021 19:36

Apply suggestions from @devmotion

4623798

Co-authored-by: David Widmann <devmotion@users.noreply.github.com>

Apply suggestions from code review

d86ab1d

Co-authored-by: David Widmann <devmotion@users.noreply.github.com>

phipsgabler approved these changes Aug 27, 2021

View reviewed changes

yebai approved these changes Aug 31, 2021

View reviewed changes

Merge branch 'master' into tor/test-utils

0f6f784

fixed tests

857b99b

fixed tests

9199004

bors Bot changed the title ~~Add TestUtils submodule~~ [Merged by Bors] - Add TestUtils submodule Sep 9, 2021

bors Bot closed this Sep 9, 2021

bors Bot deleted the tor/test-utils branch September 9, 2021 16:28

Uh oh!

Conversation

torfjelde commented Aug 23, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

devmotion Aug 23, 2021

Choose a reason for hiding this comment

Uh oh!

torfjelde Aug 23, 2021

Choose a reason for hiding this comment

Uh oh!

phipsgabler left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

torfjelde commented Aug 23, 2021

Uh oh!

phipsgabler Aug 23, 2021

Choose a reason for hiding this comment

Uh oh!

torfjelde Aug 23, 2021

Choose a reason for hiding this comment

Uh oh!

phipsgabler Aug 23, 2021

Choose a reason for hiding this comment

Uh oh!

phipsgabler Aug 23, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

torfjelde Aug 23, 2021

Choose a reason for hiding this comment

Uh oh!

phipsgabler commented Aug 23, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

devmotion left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

torfjelde commented Sep 8, 2021

Uh oh!

bors Bot commented Sep 8, 2021

Uh oh!

torfjelde commented Sep 9, 2021

Uh oh!

bors Bot commented Sep 9, 2021

Uh oh!

torfjelde commented Sep 9, 2021

Uh oh!

bors Bot commented Sep 9, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

torfjelde commented Aug 23, 2021 •

edited

Loading

phipsgabler Aug 23, 2021 •

edited

Loading

phipsgabler commented Aug 23, 2021 •

edited

Loading