index.json

[{"authors":null,"categories":null,"content":"My research is on the interface between the theory and application of Statistics. I have worked on a wide range of statistical problems ranging over air pollution, anxiety and depression, astronomy, athletics, concrete, cosmetics, flooding, fungicides, fuel filters, marketing of cars, obesity and schizophrenia. I have also served as a consultant to industry on statistical issues.\n","date":1654905600,"expirydate":-62135596800,"kind":"term","lang":"en","lastmod":1654905600,"objectID":"2525497d367e79493fd32b198b28f040","permalink":"https://julianfaraway.github.io/author/julian-faraway/","publishdate":"0001-01-01T00:00:00Z","relpermalink":"/author/julian-faraway/","section":"authors","summary":"My research is on the interface between the theory and application of Statistics. I have worked on a wide range of statistical problems ranging over air pollution, anxiety and depression, astronomy, athletics, concrete, cosmetics, flooding, fungicides, fuel filters, marketing of cars, obesity and schizophrenia.","tags":null,"title":"Julian Faraway","type":"authors"},{"authors":null,"categories":null,"content":"","date":1607990400,"expirydate":-62135596800,"kind":"section","lang":"en","lastmod":1607990400,"objectID":"","permalink":"https://julianfaraway.github.io/book/","publishdate":"0001-01-01T00:00:00Z","relpermalink":"/book/","section":"book","summary":"","tags":null,"title":"Books","type":"book"},{"authors":null,"categories":null,"content":"Flexibility This feature can be used for publishing content such as:\nOnline courses Project or software documentation Tutorials The courses folder may be renamed. For example, we can rename it to docs for software/project documentation or tutorials for creating an online course.\nDelete tutorials To remove these pages, delete the courses folder and see below to delete the associated menu link.\nUpdate site menu After renaming or deleting the courses folder, you may wish to update any [[main]] menu links to it by editing your menu configuration at config/_default/menus.toml.\nFor example, if you delete this folder, you can remove the following from your menu configuration:\n[[main]] name = \u0026quot;Courses\u0026quot; url = \u0026quot;courses/\u0026quot; weight = 50 Or, if you are creating a software documentation site, you can rename the courses folder to docs and update the associated Courses menu configuration to:\n[[main]] name = \u0026quot;Docs\u0026quot; url = \u0026quot;docs/\u0026quot; weight = 50 Update the docs menu If you use the docs layout, note that the name of the menu in the front matter should be in the form [menu.X] where X is the folder name. Hence, if you rename the courses/example/ folder, you should also rename the menu definitions in the front matter of files within courses/example/ from [menu.example] to [menu.\u0026lt;NewFolderName\u0026gt;].\n","date":1536451200,"expirydate":-62135596800,"kind":"section","lang":"en","lastmod":1536451200,"objectID":"59c3ce8e202293146a8a934d37a4070b","permalink":"https://julianfaraway.github.io/courses/example/","publishdate":"2018-09-09T00:00:00Z","relpermalink":"/courses/example/","section":"courses","summary":"Learn how to use Academic's docs layout for publishing online courses, software documentation, and tutorials.","tags":null,"title":"Overview","type":"docs"},{"authors":null,"categories":null,"content":"In this tutorial, I\u0026rsquo;ll share my top 10 tips for getting started with Academic:\nTip 1 Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis posuere tellus ac convallis placerat. Proin tincidunt magna sed ex sollicitudin condimentum. Sed ac faucibus dolor, scelerisque sollicitudin nisi. Cras purus urna, suscipit quis sapien eu, pulvinar tempor diam. Quisque risus orci, mollis id ante sit amet, gravida egestas nisl. Sed ac tempus magna. Proin in dui enim. Donec condimentum, sem id dapibus fringilla, tellus enim condimentum arcu, nec volutpat est felis vel metus. Vestibulum sit amet erat at nulla eleifend gravida.\nNullam vel molestie justo. Curabitur vitae efficitur leo. In hac habitasse platea dictumst. Sed pulvinar mauris dui, eget varius purus congue ac. Nulla euismod, lorem vel elementum dapibus, nunc justo porta mi, sed tempus est est vel tellus. Nam et enim eleifend, laoreet sem sit amet, elementum sem. Morbi ut leo congue, maximus velit ut, finibus arcu. In et libero cursus, rutrum risus non, molestie leo. Nullam congue quam et volutpat malesuada. Sed risus tortor, pulvinar et dictum nec, sodales non mi. Phasellus lacinia commodo laoreet. Nam mollis, erat in feugiat consectetur, purus eros egestas tellus, in auctor urna odio at nibh. Mauris imperdiet nisi ac magna convallis, at rhoncus ligula cursus.\nCras aliquam rhoncus ipsum, in hendrerit nunc mattis vitae. Duis vitae efficitur metus, ac tempus leo. Cras nec fringilla lacus. Quisque sit amet risus at ipsum pharetra commodo. Sed aliquam mauris at consequat eleifend. Praesent porta, augue sed viverra bibendum, neque ante euismod ante, in vehicula justo lorem ac eros. Suspendisse augue libero, venenatis eget tincidunt ut, malesuada at lorem. Donec vitae bibendum arcu. Aenean maximus nulla non pretium iaculis. Quisque imperdiet, nulla in pulvinar aliquet, velit quam ultrices quam, sit amet fringilla leo sem vel nunc. Mauris in lacinia lacus.\nSuspendisse a tincidunt lacus. Curabitur at urna sagittis, dictum ante sit amet, euismod magna. Sed rutrum massa id tortor commodo, vitae elementum turpis tempus. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Aenean purus turpis, venenatis a ullamcorper nec, tincidunt et massa. Integer posuere quam rutrum arcu vehicula imperdiet. Mauris ullamcorper quam vitae purus congue, quis euismod magna eleifend. Vestibulum semper vel augue eget tincidunt. Fusce eget justo sodales, dapibus odio eu, ultrices lorem. Duis condimentum lorem id eros commodo, in facilisis mauris scelerisque. Morbi sed auctor leo. Nullam volutpat a lacus quis pharetra. Nulla congue rutrum magna a ornare.\nAliquam in turpis accumsan, malesuada nibh ut, hendrerit justo. Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Quisque sed erat nec justo posuere suscipit. Donec ut efficitur arcu, in malesuada neque. Nunc dignissim nisl massa, id vulputate nunc pretium nec. Quisque eget urna in risus suscipit ultricies. Pellentesque odio odio, tincidunt in eleifend sed, posuere a diam. Nam gravida nisl convallis semper elementum. Morbi vitae felis faucibus, vulputate orci placerat, aliquet nisi. Aliquam erat volutpat. Maecenas sagittis pulvinar purus, sed porta quam laoreet at.\nTip 2 Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis posuere tellus ac convallis placerat. Proin tincidunt magna sed ex sollicitudin condimentum. Sed ac faucibus dolor, scelerisque sollicitudin nisi. Cras purus urna, suscipit quis sapien eu, pulvinar tempor diam. Quisque risus orci, mollis id ante sit amet, gravida egestas nisl. Sed ac tempus magna. Proin in dui enim. Donec condimentum, sem id dapibus fringilla, tellus enim condimentum arcu, nec volutpat est felis vel metus. Vestibulum sit amet erat at nulla eleifend gravida.\nNullam vel molestie justo. Curabitur vitae efficitur leo. In hac habitasse platea dictumst. Sed pulvinar mauris dui, eget varius purus congue ac. Nulla euismod, lorem vel elementum dapibus, nunc justo porta mi, sed tempus est est vel tellus. Nam et enim eleifend, laoreet sem sit amet, elementum sem. Morbi ut leo congue, maximus velit ut, finibus arcu. In et libero cursus, rutrum risus non, molestie leo. Nullam congue quam et volutpat malesuada. Sed risus tortor, pulvinar et dictum nec, sodales non mi. Phasellus lacinia commodo laoreet. Nam mollis, erat in feugiat consectetur, purus eros egestas tellus, in auctor urna odio at nibh. Mauris imperdiet nisi ac magna convallis, at rhoncus ligula cursus.\nCras aliquam rhoncus ipsum, in hendrerit nunc mattis vitae. Duis vitae efficitur metus, ac tempus leo. Cras nec fringilla lacus. Quisque sit amet risus at ipsum pharetra commodo. Sed aliquam mauris at consequat eleifend. Praesent porta, augue sed viverra bibendum, neque ante euismod ante, in vehicula justo lorem ac eros. Suspendisse augue libero, venenatis eget tincidunt ut, malesuada at lorem. Donec vitae bibendum arcu. Aenean maximus nulla non pretium iaculis. Quisque imperdiet, nulla in pulvinar aliquet, velit quam ultrices quam, sit amet fringilla leo sem vel nunc. Mauris in lacinia lacus.\nSuspendisse a tincidunt lacus. Curabitur at urna sagittis, dictum ante sit amet, euismod magna. Sed rutrum massa id tortor commodo, vitae elementum turpis tempus. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Aenean purus turpis, venenatis a ullamcorper nec, tincidunt et massa. Integer posuere quam rutrum arcu vehicula imperdiet. Mauris ullamcorper quam vitae purus congue, quis euismod magna eleifend. Vestibulum semper vel augue eget tincidunt. Fusce eget justo sodales, dapibus odio eu, ultrices lorem. Duis condimentum lorem id eros commodo, in facilisis mauris scelerisque. Morbi sed auctor leo. Nullam volutpat a lacus quis pharetra. Nulla congue rutrum magna a ornare.\nAliquam in turpis accumsan, malesuada nibh ut, hendrerit justo. Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Quisque sed erat nec justo posuere suscipit. Donec ut efficitur arcu, in malesuada neque. Nunc dignissim nisl massa, id vulputate nunc pretium nec. Quisque eget urna in risus suscipit ultricies. Pellentesque odio odio, tincidunt in eleifend sed, posuere a diam. Nam gravida nisl convallis semper elementum. Morbi vitae felis faucibus, vulputate orci placerat, aliquet nisi. Aliquam erat volutpat. Maecenas sagittis pulvinar purus, sed porta quam laoreet at.\n","date":1557010800,"expirydate":-62135596800,"kind":"page","lang":"en","lastmod":1557010800,"objectID":"74533bae41439377bd30f645c4677a27","permalink":"https://julianfaraway.github.io/courses/example/example1/","publishdate":"2019-05-05T00:00:00+01:00","relpermalink":"/courses/example/example1/","section":"courses","summary":"In this tutorial, I\u0026rsquo;ll share my top 10 tips for getting started with Academic:\nTip 1 Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis posuere tellus ac convallis placerat. Proin tincidunt magna sed ex sollicitudin condimentum.","tags":null,"title":"Example Page 1","type":"docs"},{"authors":null,"categories":null,"content":"Here are some more tips for getting started with Academic:\nTip 3 Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis posuere tellus ac convallis placerat. Proin tincidunt magna sed ex sollicitudin condimentum. Sed ac faucibus dolor, scelerisque sollicitudin nisi. Cras purus urna, suscipit quis sapien eu, pulvinar tempor diam. Quisque risus orci, mollis id ante sit amet, gravida egestas nisl. Sed ac tempus magna. Proin in dui enim. Donec condimentum, sem id dapibus fringilla, tellus enim condimentum arcu, nec volutpat est felis vel metus. Vestibulum sit amet erat at nulla eleifend gravida.\nNullam vel molestie justo. Curabitur vitae efficitur leo. In hac habitasse platea dictumst. Sed pulvinar mauris dui, eget varius purus congue ac. Nulla euismod, lorem vel elementum dapibus, nunc justo porta mi, sed tempus est est vel tellus. Nam et enim eleifend, laoreet sem sit amet, elementum sem. Morbi ut leo congue, maximus velit ut, finibus arcu. In et libero cursus, rutrum risus non, molestie leo. Nullam congue quam et volutpat malesuada. Sed risus tortor, pulvinar et dictum nec, sodales non mi. Phasellus lacinia commodo laoreet. Nam mollis, erat in feugiat consectetur, purus eros egestas tellus, in auctor urna odio at nibh. Mauris imperdiet nisi ac magna convallis, at rhoncus ligula cursus.\nCras aliquam rhoncus ipsum, in hendrerit nunc mattis vitae. Duis vitae efficitur metus, ac tempus leo. Cras nec fringilla lacus. Quisque sit amet risus at ipsum pharetra commodo. Sed aliquam mauris at consequat eleifend. Praesent porta, augue sed viverra bibendum, neque ante euismod ante, in vehicula justo lorem ac eros. Suspendisse augue libero, venenatis eget tincidunt ut, malesuada at lorem. Donec vitae bibendum arcu. Aenean maximus nulla non pretium iaculis. Quisque imperdiet, nulla in pulvinar aliquet, velit quam ultrices quam, sit amet fringilla leo sem vel nunc. Mauris in lacinia lacus.\nSuspendisse a tincidunt lacus. Curabitur at urna sagittis, dictum ante sit amet, euismod magna. Sed rutrum massa id tortor commodo, vitae elementum turpis tempus. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Aenean purus turpis, venenatis a ullamcorper nec, tincidunt et massa. Integer posuere quam rutrum arcu vehicula imperdiet. Mauris ullamcorper quam vitae purus congue, quis euismod magna eleifend. Vestibulum semper vel augue eget tincidunt. Fusce eget justo sodales, dapibus odio eu, ultrices lorem. Duis condimentum lorem id eros commodo, in facilisis mauris scelerisque. Morbi sed auctor leo. Nullam volutpat a lacus quis pharetra. Nulla congue rutrum magna a ornare.\nAliquam in turpis accumsan, malesuada nibh ut, hendrerit justo. Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Quisque sed erat nec justo posuere suscipit. Donec ut efficitur arcu, in malesuada neque. Nunc dignissim nisl massa, id vulputate nunc pretium nec. Quisque eget urna in risus suscipit ultricies. Pellentesque odio odio, tincidunt in eleifend sed, posuere a diam. Nam gravida nisl convallis semper elementum. Morbi vitae felis faucibus, vulputate orci placerat, aliquet nisi. Aliquam erat volutpat. Maecenas sagittis pulvinar purus, sed porta quam laoreet at.\nTip 4 Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis posuere tellus ac convallis placerat. Proin tincidunt magna sed ex sollicitudin condimentum. Sed ac faucibus dolor, scelerisque sollicitudin nisi. Cras purus urna, suscipit quis sapien eu, pulvinar tempor diam. Quisque risus orci, mollis id ante sit amet, gravida egestas nisl. Sed ac tempus magna. Proin in dui enim. Donec condimentum, sem id dapibus fringilla, tellus enim condimentum arcu, nec volutpat est felis vel metus. Vestibulum sit amet erat at nulla eleifend gravida.\nNullam vel molestie justo. Curabitur vitae efficitur leo. In hac habitasse platea dictumst. Sed pulvinar mauris dui, eget varius purus congue ac. Nulla euismod, lorem vel elementum dapibus, nunc justo porta mi, sed tempus est est vel tellus. Nam et enim eleifend, laoreet sem sit amet, elementum sem. Morbi ut leo congue, maximus velit ut, finibus arcu. In et libero cursus, rutrum risus non, molestie leo. Nullam congue quam et volutpat malesuada. Sed risus tortor, pulvinar et dictum nec, sodales non mi. Phasellus lacinia commodo laoreet. Nam mollis, erat in feugiat consectetur, purus eros egestas tellus, in auctor urna odio at nibh. Mauris imperdiet nisi ac magna convallis, at rhoncus ligula cursus.\nCras aliquam rhoncus ipsum, in hendrerit nunc mattis vitae. Duis vitae efficitur metus, ac tempus leo. Cras nec fringilla lacus. Quisque sit amet risus at ipsum pharetra commodo. Sed aliquam mauris at consequat eleifend. Praesent porta, augue sed viverra bibendum, neque ante euismod ante, in vehicula justo lorem ac eros. Suspendisse augue libero, venenatis eget tincidunt ut, malesuada at lorem. Donec vitae bibendum arcu. Aenean maximus nulla non pretium iaculis. Quisque imperdiet, nulla in pulvinar aliquet, velit quam ultrices quam, sit amet fringilla leo sem vel nunc. Mauris in lacinia lacus.\nSuspendisse a tincidunt lacus. Curabitur at urna sagittis, dictum ante sit amet, euismod magna. Sed rutrum massa id tortor commodo, vitae elementum turpis tempus. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Aenean purus turpis, venenatis a ullamcorper nec, tincidunt et massa. Integer posuere quam rutrum arcu vehicula imperdiet. Mauris ullamcorper quam vitae purus congue, quis euismod magna eleifend. Vestibulum semper vel augue eget tincidunt. Fusce eget justo sodales, dapibus odio eu, ultrices lorem. Duis condimentum lorem id eros commodo, in facilisis mauris scelerisque. Morbi sed auctor leo. Nullam volutpat a lacus quis pharetra. Nulla congue rutrum magna a ornare.\nAliquam in turpis accumsan, malesuada nibh ut, hendrerit justo. Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Quisque sed erat nec justo posuere suscipit. Donec ut efficitur arcu, in malesuada neque. Nunc dignissim nisl massa, id vulputate nunc pretium nec. Quisque eget urna in risus suscipit ultricies. Pellentesque odio odio, tincidunt in eleifend sed, posuere a diam. Nam gravida nisl convallis semper elementum. Morbi vitae felis faucibus, vulputate orci placerat, aliquet nisi. Aliquam erat volutpat. Maecenas sagittis pulvinar purus, sed porta quam laoreet at.\n","date":1557010800,"expirydate":-62135596800,"kind":"page","lang":"en","lastmod":1557010800,"objectID":"1c2b5a11257c768c90d5050637d77d6a","permalink":"https://julianfaraway.github.io/courses/example/example2/","publishdate":"2019-05-05T00:00:00+01:00","relpermalink":"/courses/example/example2/","section":"courses","summary":"Here are some more tips for getting started with Academic:\nTip 3 Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis posuere tellus ac convallis placerat. Proin tincidunt magna sed ex sollicitudin condimentum.","tags":null,"title":"Example Page 2","type":"docs"},{"authors":[],"categories":null,"content":" Click on the Slides button above to view the built-in slides feature. Slides can be added in a few ways:\nCreate slides using Academic\u0026rsquo;s Slides feature and link using slides parameter in the front matter of the talk file Upload an existing slide deck to static/ and link using url_slides parameter in the front matter of the talk file Embed your slides (e.g. Google Slides) or presentation video on this page using shortcodes. Further talk details can easily be added to this page using Markdown and $\\rm \\LaTeX$ math code.\n","date":1906549200,"expirydate":-62135596800,"kind":"page","lang":"en","lastmod":1906549200,"objectID":"96344c08df50a1b693cc40432115cbe3","permalink":"https://julianfaraway.github.io/talk/example/","publishdate":"2017-01-01T00:00:00Z","relpermalink":"/talk/example/","section":"talk","summary":"An example talk using Academic's Markdown slides feature.","tags":[],"title":"Example Talk","type":"talk"},{"authors":[],"categories":[],"content":"Two years ago, I converted my lecture notes to Bookdown. I have been following the development of Quarto with interest. Although Bookdown is not going away, it does seem that future developments will be based on Quarto. For this reason, I decided to convert my lecture notes to Quarto. Here\u0026rsquo;s my experience.\nI installed Quarto. I started with a conversion guide written by Stefanie Butland and Ileana Fenwick. I renamed by Rmd files to qmd files and moved some information around in a couple of config files as described in the guide. Within about 20 minutes, I had the whole book rendered. Great - how easy was that? Unfortunately, on examining the book, I realised I had more work to do.\nThe first problem was with the displayed equations. Bookdown could be finicky about the placement of the delimiting $$\u0026rsquo;s and my labels were all inside the delimiters. Quarto is more fussy about the layout - you need something like this:\n$$ \\your \\latex \\stuff \\here $$ {#eq-yourlabel} In particular, you need to have the $$\u0026rsquo;s on a new line. Fortunately, getting the equations to preview within the editor was a reliable indication that I had done it right - there was no need to keep recompiling to check my work.\nThe second problem was the cross referencing. Previously, for a figure, you would have the label in the {r myfig} chunk and refer to it as Figure \\@ref(fig:myfig) but this becomes:\n```{r} #| label: fig-myfig plot(x,y) ``` and is referred to as @fig-myfig which will render as something like Figure 5.1 so you\u0026rsquo;ll need to delete the extra \u0026ldquo;Figure \u0026quot; from your text. You will need to change the equation, table, section etc. crossreferences also.\nThe third problem required me to reformat some R chunks. As seen above, the desired format for the option in R chunks has changed to have these each listed on a seperate line, prefixed with a #|. This does make sense and the old format still works - sometimes. But if you have some figures with multiple plots or other features, you will have to convert. The new multicolumn setup is nice but you will need to do some editing.\nPreviously, Bookdown allowed you to load some R code before running each chapter using before_chapter_script: but I could not get this to work in Quarto. I had to put this in the .Rprofile to get this working (I tend to avoid using .Rprofile for reproducibility reasons).\nSomeone with better scripting skills than me could write a conversion script to deal with many of these problems (although some hand-editing would still be necessary). I don\u0026rsquo;t think anyone has done this yet.\nAlthough it took a bit more work than I had initially thought, I am pleased with the result. The design of Quarto is better than Bookdown and it already has some nice features that I want to try. I am also looking forward to a more consistent syntax across the other R markdown documents such as presentations, websites and scripts that I commonly use.\n","date":1660694400,"expirydate":-62135596800,"kind":"page","lang":"en","lastmod":1660728895,"objectID":"fb7ff69a661d9ff8a715ac8bd441f801","permalink":"https://julianfaraway.github.io/post/converting-from-bookdown-to-quarto/","publishdate":"2022-08-17T00:00:00Z","relpermalink":"/post/converting-from-bookdown-to-quarto/","section":"post","summary":"Two years ago, I converted my lecture notes to Bookdown. I have been following the development of Quarto with interest. Although Bookdown is not going away, it does seem that future developments will be based on Quarto.","tags":[],"title":"Converting from Bookdown to Quarto","type":"post"},{"authors":["Julian Faraway","James Boxall-Clasby","Edward J. Feil","Marjorie Gibbon","Oliver Hatfield","Barbara Kasprzyk-Hordern","Theresa Smith"],"categories":null,"content":"","date":1654905600,"expirydate":-62135596800,"kind":"page","lang":"en","lastmod":1654905600,"objectID":"7589a55d0546caf9558cf765d0823282","permalink":"https://julianfaraway.github.io/publication/challengeswbe/","publishdate":"2022-06-11T00:00:00Z","relpermalink":"/publication/challengeswbe/","section":"publication","summary":"Researchers around the world have demonstrated correlations between measurements of SARS-CoV-2 RNA in wastewater (WW) and case rates of COVID-19 derived from direct testing of individuals. This has raised concerns that wastewater-based epidemiology (WBE) methods might be used to quantify the spread of this and other diseases, perhaps faster than direct testing, and with less expense and intrusion. We illustrate, using data from Scotland and the USA, the issues regarding the construction of effective predictive models for disease case rates. We discuss the effects of variation in, and the problem of aligning, public health (PH) reporting and WW measurements. We investigate time-varying effects in PH-reported case rates and their relationship to WW measurements. We show the lack of proportionality of WW measurements to case rates with associated spatial heterogeneity. We illustrate how the precision of predictions is affected by the level of aggregation chosen. We determine whether PH or WW measurements are the leading indicators of disease and how they may be used in conjunction to produce predictive models. The prospects of using WW-based predictive models with or without ongoing PH data are discussed.","tags":null,"title":"Challenges in realising the potential of wastewater-based epidemiology to quantitatively monitor and predict the spread of disease","type":"publication"},{"authors":null,"categories":null,"content":"I am working with Barbara Kasprzyk-Hordern [on building an Early Warning System for community-wide infectious disease spread](UKRI GCRF/Newton Fund : Building an Early Warning System for community-wide infectious disease spread: SARS-CoV-2 tracking in Africa via environment fingerprinting- COVID-19). I am also involved in EDGE which is environment fingerprinting via digital technology - a new paradigm in hazard forecasting and early-warning systems for health risks in Africa.\n","date":1608508800,"expirydate":-62135596800,"kind":"page","lang":"en","lastmod":1608508800,"objectID":"dbca0cd1f8cdd3900ba31f9a01a489b5","permalink":"https://julianfaraway.github.io/project/wastewater/","publishdate":"2020-12-21T00:00:00Z","relpermalink":"/project/wastewater/","section":"project","summary":"Sampling wastewater for Covid-19 and environmental monitoring","tags":["research"],"title":"Wastewater Monitoring","type":"project"},{"authors":null,"categories":null,"content":"","date":1607990400,"expirydate":-62135596800,"kind":"page","lang":"en","lastmod":1607990400,"objectID":"0934190410f24512dc3784a34e6a9407","permalink":"https://julianfaraway.github.io/book/glmm/","publishdate":"2020-12-15T00:00:00Z","relpermalink":"/book/glmm/","section":"book","summary":"Code, Errata and Supplementary Materials","tags":["book"],"title":"Extending the Linear Model with R","type":"book"},{"authors":null,"categories":null,"content":"","date":1607990400,"expirydate":-62135596800,"kind":"page","lang":"en","lastmod":1607990400,"objectID":"b8ccd8d29e2d829b2ad3643800f89bac","permalink":"https://julianfaraway.github.io/book/lmp/","publishdate":"2020-12-15T00:00:00Z","relpermalink":"/book/lmp/","section":"book","summary":"Code, Errata and Supplementary Materials","tags":["book"],"title":"Linear Models with Python","type":"book"},{"authors":null,"categories":null,"content":"","date":1607990400,"expirydate":-62135596800,"kind":"page","lang":"en","lastmod":1607990400,"objectID":"280fed2d5c9a4fa25b7e2ebaf107acfb","permalink":"https://julianfaraway.github.io/book/lmr/","publishdate":"2020-12-15T00:00:00Z","relpermalink":"/book/lmr/","section":"book","summary":"Code, Errata and Supplementary Materials","tags":["book"],"title":"Linear Models with R","type":"book"},{"authors":null,"categories":null,"content":"I have a EPSRC GCRF grant for statistics training in Mongolia. We have a particular interest in the effects of air pollution on health. We delivered a workshop in Ulaanbaatar in May 2019. You can see the training materials and read more in the report. I have been working with Andreas Kyprianou who has longstanding connections to Mongolia.\n","date":1607990400,"expirydate":-62135596800,"kind":"page","lang":"en","lastmod":1607990400,"objectID":"79eefc8813bdb7249d53a25493ac5d5b","permalink":"https://julianfaraway.github.io/project/mongolia/","publishdate":"2020-12-15T00:00:00Z","relpermalink":"/project/mongolia/","section":"project","summary":"Statistics and Data Science training to build research capacity to address societal problems in Mongolia","tags":["research"],"title":"Statistics training in Mongolia","type":"project"},{"authors":[],"categories":[],"content":"When I heard that the UK government had lost 16,000 covid-19 cases meaning that the contacts of infected persons were not traced and informed, I got angry. The problem was attributed to an Excel \u0026ldquo;glitch\u0026rdquo; so I posted a rather caustic tweet about how Excel should not be used for data analysis. Twitter is a quickfire medium and not the place for nuance. Here\u0026rsquo;s my more moderate view on the matter:\nIn a long career, I\u0026rsquo;ve spent a lot of time cleaning data delivered to me in Excel format. Although it is possible to maintain data cleanly in Excel, this is often not the case. It often takes a lot of time and frustration to restore the data into a form suitable for statistical analysis. Although one could blame the producers of the data, Excel gives them more than enough freedom to mess things up. In particular, there\u0026rsquo;s no clear boundary in Excel between data and analysis - it\u0026rsquo;s all mixed together in a single spreadsheet. It\u0026rsquo;s this experience that makes me angry when I hear about Excel data \u0026ldquo;glitches\u0026rdquo;. Please use almost any other format.\nExcel is installed on a large proportion of computers and a very large number of people have some experience of using it. It is very versatile software. It\u0026rsquo;s like a swiss army knife, performing a wide range of tasks, often in adequate manner without excelling (hah!) at any particular task. It\u0026rsquo;s been so successful because it so useful. Unfortunately, people sometimes fail to recognise when Excel is not the right tool. They are familiar with Excel and are either unaware of or unwilling to learn a more appropriate software tool.\nMy tweet about the missing covid cases was inaccurate. I said that Excel is not good for (statistical) data analysis when I should have said that Excel is not good for database management. Both are true but in this instance, the error was caused by using Excel in the data handling pipeline. This was a database problem, not a statistics problem.\nIt\u0026rsquo;s all too easy to point the finger at people making errors - how could you be so stupid? etc. But all of us make errors because we are human. Our defence against these errors is using systems that are designed to catch errors. In the missing covid cases example, Excel could have been used successfully. Some people might say that the users and not Excel were at fault. But this is the principal weakness of Excel in comparison to purpose-built statistical software or database management software. These latter software systems make it much easier to build in the error checking, the auditing and the reproducibility that are essential for the minimisation of human error.\n","date":1601942400,"expirydate":-62135596800,"kind":"page","lang":"en","lastmod":1602005090,"objectID":"737daef81529e4b762be886ee5effb2a","permalink":"https://julianfaraway.github.io/post/a-more-moderate-view-of-excel/","publishdate":"2020-10-06T00:00:00Z","relpermalink":"/post/a-more-moderate-view-of-excel/","section":"post","summary":"When I heard that the UK government had lost 16,000 covid-19 cases meaning that the contacts of infected persons were not traced and informed, I got angry. The problem was attributed to an Excel \u0026ldquo;glitch\u0026rdquo; so I posted a rather caustic tweet about how Excel should not be used for data analysis.","tags":[],"title":"A more moderate view of Excel","type":"post"},{"authors":[],"categories":[],"content":"Why do this? In 2018, the UK government passed the The Public Sector Bodies (Websites and Mobile Applications) Accessibility Regulations which requires Universities (among others) to provide all website information in an \u0026ldquo;accessible\u0026rdquo; format. The UK government has provided a more detailed set of recommendations on how to meet these requirements. From the mathematical perspective, you can find a discussion of the requirements by Matthew Towers of UCL.\nIn our case, the primary beneficiaries are people with impaired vision but does include other categories of disability. In the past, we were able to meet the requirements of prior equalities legislation by providing alternative formats to affected students. This might be as simple as producing large print versions but sometimes needed more unusual formats. I was able to send my LaTeX source to a specialist who produced the required formats on a case-by-case basis depending on the students in the class. But under the new legislation, this is no longer acceptable and the primary version of my online materials must be accessible. Since some of these materials are created during the teaching period, I need to be able to create them myself.\nThe main problem for anyone teaching a mathematical subject is that the primary medium for materials is LaTeX compiled to a PDF. Despite some efforts to make LaTeX-\u0026gt;PDF accessible, it falls far short of the legal requirements. The main advantage of a webpage or an e-reader is that the fonts can be resized or changed and the page can be resized with the text being reflowed to fit. You cannot do this with a PDF. The main challenge is to convert to a format that allows this resizing and reflowing. There are other requirements for accessibility such as alternate text for images but we\u0026rsquo;ll focus on the resizing and reflowing problem.\nBookdown A big step towards the solution is to use Bookdown. You can find numerous examples at bookdown.org. In particular, bookdown: Authoring Books and Technical Documents with R Markdown is very helpful. The books shown on bookdown.org display nicely on webpages but also can be printed in a relatively standard and compact format. If you have never used R and do not want to use R, don\u0026rsquo;t worry, you don\u0026rsquo;t need to. You can use bookdown without any Statistics or R - here\u0026rsquo;s an example of some algebra lecture notes. It certainly helps if you are familiar with markdown but it should not provide much challenge to anyone who has tackled LaTeX.\nConversion Process I won\u0026rsquo;t attempt to provide a step-by-step guide here. Start with the first two chapters of the bookdown book to get going. It would be a lot easier to write the bookdown lecture notes directly but you may have a large amount of existing material that needs to be converted from LaTeX. Some of this will be documents of a few pages such as exercise sheets and solutions. You don\u0026rsquo;t need bookdown for these (we\u0026rsquo;ll discuss these later). Bookdown is best for a longer document with multiple sections or chapters.\nYou don\u0026rsquo;t need to use Rstudio but it does have some useful features. In particular, making a new bookdown project is nice because it creates all the necessary setup files. It also has a nice previewer. But Rstudio was not built for editing LaTeX and I missed all my usual keyboard shortcuts.\nMathjax takes care of creating all the mathematical equations, both displayed and inline. It\u0026rsquo;s important to understand this is not done using the usual LaTeX compilation process. I had a lot of LaTeX macros from my predecessor throughout the lecture notes. Mathjax didn\u0026rsquo;t understand these and I had to replace them with standard LaTeX. Mathjax only understands a limited set of LaTeX packages. For example, it does not have boldmath but did have boldsymbol so I needed to make some translations.\nPandoc is wonderful. It will be installed as part of the bookdown installation process. It does an excellent job of converting your LaTeX into markdown. But it fails on the referencing and citation system used by bookdown. I had to fix all these manually.\nThe knitr engine is usually used for processing R code chunks but I was delighted to find that it also handles Tikz.\nAlthough you can use bookdown for short LaTeX documents, it\u0026rsquo;s easier to use R markdown (on which bookdown is based). Suppose you have exercise.tex, convert this to markdown with:\npandoc -o exercise.md exercise.tex You may need to fix up the markdown. Add a YAML such as:\n--- title: \u0026quot;Exercise One\u0026quot; author: \u0026quot;Julian Faraway\u0026quot; date: \u0026quot;01/09/2020\u0026quot; output: html_document --- Change the name to exercise.Rmd and convert to HTML with:\nRscript -e \u0026quot;rmarkdown::render('exercise.Rmd')\u0026quot; You can produce PDF output from the same source. Let\u0026rsquo;s suppose this is for students who want to print out some hard copy. As a rule, they will want a compact format so that there are less pages to print. I put documentclass: article geometry: \u0026quot;margin=1.5cm,nohead\u0026quot; fontsize: 10pt in the YAML header of index.Rmd. Even so, the output was about 15% longer than the original PDF version. I could cut that down a bit with further fiddling but I don\u0026rsquo;t want to be messing with LaTeX floats. You can also create an epub version for e-readers although I\u0026rsquo;m not sure Kindles and the like can cope with Mathjax. R Markdown can also be output in Microsoft Word format which many mathematicians would regard as the ultimate sacrilege.\nI have yet to find a convenient way to add alternative text to images using bookdown. The alt-text is confounded with the fig-cap but these should be two different texts. I am also unsure what alt-text is best for something like a scatterplot. Conclusion It took some effort to understand how bookdown works and make the conversion. In the future, writing documents in markdown with embeded LaTeX maths will be no more difficult than creating a full LaTeX document. Given the markdown is simpler than LaTeX, I expect it will be easier. It also allows some more interesting possibilities.\nI was pleased with resulting bookdown lecture notes. They are easier to read on screen than a PDF given the flexibility in resizing and reflowing the content. Although I used to provide students with printed lecture notes, I noticed that many used the online PDF version in preference. Although the conversion was motivated by the need to satisfy accessibility legislation, I think the result will benefit all students. Now that more instruction is online, the primary version of the materials should be the web version rather than the hard copy version.\n","date":1598918400,"expirydate":-62135596800,"kind":"page","lang":"en","lastmod":1598970544,"objectID":"80a2a7dffef5ad8b47a58143a2db22b1","permalink":"https://julianfaraway.github.io/post/converting-to-accessible-lecture-notes/","publishdate":"2020-09-01T00:00:00Z","relpermalink":"/post/converting-to-accessible-lecture-notes/","section":"post","summary":"Why do this? In 2018, the UK government passed the The Public Sector Bodies (Websites and Mobile Applications) Accessibility Regulations which requires Universities (among others) to provide all website information in an \u0026ldquo;accessible\u0026rdquo; format.","tags":[],"title":"Converting to Accessible Lecture Notes","type":"post"},{"authors":null,"categories":null,"content":"","date":1598572800,"expirydate":-62135596800,"kind":"page","lang":"en","lastmod":1598572800,"objectID":"8ac9fb0b4dcbf577115436a939cf54ec","permalink":"https://julianfaraway.github.io/book/brinla/","publishdate":"2020-08-28T00:00:00Z","relpermalink":"/book/brinla/","section":"book","summary":"Code and Errata","tags":["book"],"title":"Bayesian Regression Modeling with INLA","type":"book"},{"authors":[],"categories":[],"content":"As anyone in the UK knows, A-level exams, taken at age 18 and mostly used to determined admission to university were cancelled this year due to the pandemic. The UK government asked OFQUAL (a semi-independent government body) to determine the A-level grades. OFQUAL developed an algorithm based on past school performance and teacher rankings of students explained here. When these results were released there was an uproar because large numbers of students did not receive the grades they were expecting and rejected from their chosen universities. After days of pressure the UK government backtracked on these predicted results and also allowed teacher predicted grades. For many universities, this has resulted in more students having qualified to enter than they can accommodate. Hence chaos.\nNaturally statisticians were very interested in this algorithm. Back at the beginning of the process, the Royal Statistical Society had volunteered professorial help in constructing the algorithm. But OFQUAL insisted on a highly restrictive NDA to which the RSS declined to agree. After details of the algorithm emerged, Guy Nason described several deficiencies in the algorithm. The algorithm had some biases that favoured small independent schools (that usually have wealthier students) and tended to mark down larger state schools (that usually have less privileged students). My colleague in Computer Science at Bath, Tom Haines, describes other problems with the construction of the algorithm (although I do take umbrage at him blaming statisticians for this! – we did try to help)\nStatisticians have suggested improvements to the algorithm that would avoid some of the bias problems but given the information available, students would still have been upset at the result. Given that the school of the student was one of the few useful predictors available, one could not avoid using this information and yet the very fact that this was done was found highly objectionable by many. Why should the school you attend determine your university outcome? Furthermore, due to a natural phenomenom known as “regression to the mean”, predictions for students who had good reasons to expect to do well will be shifted downwards. Even had OFQUAL taken more professional advice, many students would still have been angry about the results and the media uproar would have been much the same.\nI propose that an entirely different approach should have been used. Few really care what A-level grades they get – they care which university will accept them. We can issue a pandemic certificate of completion for the A-levels and deal with the university admission problem directly. Here’s my proposal:\nWait until all universities have made their offers and students have made their firm (first) choice and insurance (second) choice. Oxbridge is at the top of the tree and (as usual) pick first. They randomly choose students from among those they have accepted. They would want to control numbers on different courses and in different colleges but they must make a random selection. They pass on their rejected students to the next tier of universities. As in a normal year, the next tier of universities would wait until they receive their insurance students from Oxbridge. This year they will accept all the insurance students and randomly fill their remaining places with students who held them as first choice. They pass on their rejected students to the next tier as in a normal year. The process repeats until all students have been (randomly) allocated. The sequence of universities in the decision process is determined by the entry tariff for the given subject as would happen in a normal year. The only differences are that the selection is random and all insurance students are accepted. Now there would be need to be some modifications occasionally if capacity constraints are hit or for other uncommon circumstances. Given that there is about enough capacity in the university system as a whole for all students, my proposed algorithm would ensure that most students receive their first or second choice of university. Now doubtless there would be some sad face photos of students whose hopes and dreams have been crushed by not going to their first choice of university and having to suffer through the horrors of their second choice. But the important difference is that their misfortune will be just that – bad luck and cannot be attributed to some perceived bias against them. You can’t get angry at bad luck.\nEfficiency is the other consideration as we want to allocate students to universities commensurate with their ability. Under my scheme, students chose universities where they believed they could gain admission and universities had accepted them. Given we have no A-level exam information, this is the best we can do.\nUnfortunately, it’s too late to execute this scheme as it would be politically unacceptable that already accepted students could now be rejected.\n","date":1597795200,"expirydate":-62135596800,"kind":"page","lang":"en","lastmod":1598624921,"objectID":"c0b1e01fbd6a5b5c4ab86c2f57c0a0c6","permalink":"https://julianfaraway.github.io/post/an-alternative-to-the-ofqual-algorithm/","publishdate":"2020-08-19T00:00:00Z","relpermalink":"/post/an-alternative-to-the-ofqual-algorithm/","section":"post","summary":"As anyone in the UK knows, A-level exams, taken at age 18 and mostly used to determined admission to university were cancelled this year due to the pandemic. The UK government asked OFQUAL (a semi-independent government body) to determine the A-level grades.","tags":[],"title":"An Alternative to the OFQUAL algorithm","type":"post"},{"authors":[],"categories":[],"content":"Having run a couple of workshops introducing R in developing countries, I offer four pieces of advice:\nDon’t rely on the internet. The internet reaches everywhere but what may be adequate for light use by a single user quickly crumbles when large numbers try to make downloads simultaneously. We anticipated this would be a problem and came equipped with R and Rstudio on many cheap memory sticks but this is not sufficient. For example, try installing the Tidyverse and knitting an R markdown document. You will find this requires installing several packages with multiple dependencies. You’ve probably forgotten the multiple packages, pieces of software and configurations you did to make things work on your own machine. It was all so easy when you had a good internet connection. Find an old laptop with a fresh OS install and disconnect it from the internet. Discover what it takes to get it running on your R teaching materials and then you’ll be prepared. Purchase a local data SIM for that country so you can create a local WiFi hotspot. At least you’ll have a chance to access the internet in case of need (quite likely!). Also plan on a period of chaos at the beginning of the workshop to take care of the many install problems. Tidyverse. R is difficult, particularly for users only familiar with GUI-based statistics software. A tidyverse-only approach greatly simplifies the range of syntax and commands that the workshop participants will need to understand. You can create exercises that they can successfully complete. The tidyverse is sufficiently powerful that you can perform a wide range of practically useful tasks. The participants will gain a feeling of accomplishment and some ability to use R for their own work. Soon enough they will also need to learn to use base R. But if you start with base R, the entry cost is much higher and some of your students will not make it. Simplicity. This comes in two forms. Firstly, in countries where English is not the native language, you will find that most professional people (who are attending your workshop) know at least some English but that does not mean all of them are entirely fluent. In your presentation, speak slowly and enunciate your words clearly. Avoid colloquial expressions and figures of speech. Do not use complex words. As a native speaker of English, you will find this difficult. Use written documentation to supplement your spoken presentation. Consider using a translator. Secondly, consider simplicity in your R presentation. It is tempting to include some cool R tricks but beginners won’t enjoy this. Stick to the basics and reduce complexity where possible. Local Data. All too many expositions of R use overworn example datasets such as mtcars. Find some datasets from the country of the workshop. The participants will find this far more interesting and will suggest different ways to analyze the data. This will demonstrate how R can be used to turn data into knowledge. ","date":1589846400,"expirydate":-62135596800,"kind":"page","lang":"en","lastmod":1589846400,"objectID":"8255093d641ff59d9fb8fb471ca9223f","permalink":"https://julianfaraway.github.io/post/experience-from-running-r-workshops-in-developing-countries/","publishdate":"2020-05-19T00:00:00Z","relpermalink":"/post/experience-from-running-r-workshops-in-developing-countries/","section":"post","summary":"Having run a couple of workshops introducing R in developing countries, I offer four pieces of advice:\nDon’t rely on the internet. The internet reaches everywhere but what may be adequate for light use by a single user quickly crumbles when large numbers try to make downloads simultaneously.","tags":[],"title":"Experience from running R workshops in developing countries","type":"post"},{"authors":null,"categories":null,"content":"We visited in Asuncion in 2018 to deliver a statistics workshop. We visited again in 2019 to plan further collaboration. We are working with Instituto Desarrollo with support from the World Bank on a project to iqmprove the quality of production and use of administrative records for evidence-based policy formulation to support the following Sustainable Development Goals.\n","date":1576368000,"expirydate":-62135596800,"kind":"page","lang":"en","lastmod":1576368000,"objectID":"9e5ca57064cb1fe445c15dbdd20fc913","permalink":"https://julianfaraway.github.io/project/paraguay/","publishdate":"2019-12-15T00:00:00Z","relpermalink":"/project/paraguay/","section":"project","summary":"Using statistics in government","tags":["research"],"title":"Government Statistics in Paraguay","type":"project"},{"authors":["Aoibheann Brady","Julian Faraway","Ilaria Prosdocimi"],"categories":null,"content":"","date":1561420800,"expirydate":-62135596800,"kind":"page","lang":"en","lastmod":1561420800,"objectID":"09bea6e5cbc6bdfe24408ed0e26f3eb4","permalink":"https://julianfaraway.github.io/publication/peak-river-flows/","publishdate":"2017-01-01T00:00:00Z","relpermalink":"/publication/peak-river-flows/","section":"publication","summary":"We investigate the evidence for changes in the magnitude of peak river flows in Great Britain. We focus on a set of 117 near-natural “benchmark” catchments to detect trends not driven by land use and other human impacts, and aim to attribute trends in peak river flows to some climate indices such as the North Atlantic Oscillation (NAO) and the East Atlantic (EA) index. We propose modelling all stations together in a Bayesian multilevel framework to be better able to detect any signal that is present in the data by pooling information across several stations. This approach leads to the detection of a clear countrywide time trend. Additionally, in a univariate approach, both the EA and NAO indices appear to have a considerable association with peak river flows. When a multivariate approach is taken to unmask the collinearity between climate indices and time, the association between NAO and peak flows disappears, while the association with EA remains clear. This demonstrates the usefulness of a multivariate and multilevel approach when it comes to accurately attributing trends in peak river flows.","tags":null,"title":"Attribution of long-term changes in peak river flows in Great Britain","type":"publication"},{"authors":[],"categories":[],"content":"Create slides in Markdown with Academic Academic | Documentation\nFeatures Efficiently write slides in Markdown 3-in-1: Create, Present, and Publish your slides Supports speaker notes Mobile friendly slides Controls Next: Right Arrow or Space Previous: Left Arrow Start: Home Finish: End Overview: Esc Speaker notes: S Fullscreen: F Zoom: Alt + Click PDF Export: E Code Highlighting Inline code: variable\nCode block:\nporridge = \u0026quot;blueberry\u0026quot; if porridge == \u0026quot;blueberry\u0026quot;: print(\u0026quot;Eating...\u0026quot;) Math In-line math: $x + y = z$\nBlock math:\n$$ f\\left( x \\right) = ;\\frac{{2\\left( {x + 4} \\right)\\left( {x - 4} \\right)}}{{\\left( {x + 4} \\right)\\left( {x + 1} \\right)}} $$\nFragments Make content appear incrementally\n{{% fragment %}} One {{% /fragment %}} {{% fragment %}} **Two** {{% /fragment %}} {{% fragment %}} Three {{% /fragment %}} Press Space to play!\nOne **Two** Three A fragment can accept two optional parameters:\nclass: use a custom style (requires definition in custom CSS) weight: sets the order in which a fragment appears Speaker Notes Add speaker notes to your presentation\n{{% speaker_note %}} - Only the speaker can read these notes - Press `S` key to view {{% /speaker_note %}} Press the S key to view the speaker notes!\nOnly the speaker can read these notes Press S key to view Themes black: Black background, white text, blue links (default) white: White background, black text, blue links league: Gray background, white text, blue links beige: Beige background, dark text, brown links sky: Blue background, thin dark text, blue links night: Black background, thick white text, orange links serif: Cappuccino background, gray text, brown links simple: White background, black text, blue links solarized: Cream-colored background, dark green text, blue links Custom Slide Customize the slide style and background\n{{\u0026lt; slide background-image=\u0026quot;/media/boards.jpg\u0026quot; \u0026gt;}} {{\u0026lt; slide background-color=\u0026quot;#0000FF\u0026quot; \u0026gt;}} {{\u0026lt; slide class=\u0026quot;my-style\u0026quot; \u0026gt;}} Custom CSS Example Let\u0026rsquo;s make headers navy colored.\nCreate assets/css/reveal_custom.css with:\n.reveal section h1, .reveal section h2, .reveal section h3 { color: navy; } Questions? Ask\nDocumentation\n","date":1549324800,"expirydate":-62135596800,"kind":"page","lang":"en","lastmod":1549324800,"objectID":"0e6de1a61aa83269ff13324f3167c1a9","permalink":"https://julianfaraway.github.io/slides/example/","publishdate":"2019-02-05T00:00:00Z","relpermalink":"/slides/example/","section":"slides","summary":"An introduction to using Academic's Slides feature.","tags":[],"title":"Slides","type":"slides"},{"authors":null,"categories":null,"content":"Depression and anxiety are set to be leading causes of disability in high income countries by 2030, and currently cost the UK economy £9 billion a year. Improving access to psychological therapies (IAPT) is a national programme aimed at reducing this disability burden by increasing the availability of ‘talking therapies’ on the NHS. I am working with Kate Button and PhD student, Clarissa Bauer-Staub on a project to improve the allocation of psychological treaments for depression. We are working with Bath company, Mayden. See our preprint on the effects of Covid-19 on IAPT.\n","date":1545436800,"expirydate":-62135596800,"kind":"page","lang":"en","lastmod":1545436800,"objectID":"16b98aaa8d7b7499b2f84d48b67cc4bd","permalink":"https://julianfaraway.github.io/project/depression/","publishdate":"2018-12-22T00:00:00Z","relpermalink":"/project/depression/","section":"project","summary":"Using healthcare records to optimise treatments","tags":["research"],"title":"Anxiety and Depression","type":"project"},{"authors":null,"categories":null,"content":"I am working with Carroll-Ann Trotman. We use facial motion capture to model patients with cleft lip/palate or facial paralysis. Here is a list of my publications in this area.\n","date":1545350400,"expirydate":-62135596800,"kind":"page","lang":"en","lastmod":1545350400,"objectID":"d95d2c356f147c6a52c7406d43b86a22","permalink":"https://julianfaraway.github.io/project/facial/","publishdate":"2018-12-21T00:00:00Z","relpermalink":"/project/facial/","section":"project","summary":"Cleft Lip/Palate and Facial Paralysis","tags":["research"],"title":"Statistial Analysis of Facial Motion","type":"project"},{"authors":[],"categories":[],"content":"I have translated the R code in Linear Models with R into Python. The code is available as Jupyter Notebooks.\nI was able to translate most of the content into Python. Sometimes the output is similar but not the same. Python has far less statistics functionality than R but it seems most of the functionality in base R can be found in Python. R now has over 10,000 packages. Python has about ten times as many but most of these are unrelated to Statistics. My book does not depend heavily on additional packages so this was not so much of an obstacle for me. In a few cases, I rely on R packages that do not exist in Python. Doubtless a Python equivalent could be created but that will take some effort.\nAfter this experience, I can say that R is a better choice for Statistics than Python. Nevertheless, there are good reasons why one might choose to do Statistics with Python. One obvious reason is that if you already know Python, you will be reluctant to also learn R. In the UK, Python is now being taught in schools and we will soon have a wave of students who will come to university knowing Python. Python usage is also more common in several areas such as Computer Science and Engineering. Another reason to use Python is the huge range of packages ranging from text, image and signal processing to machine learning and optimisation. These go far beyond what can found in R. The Python userbase is much larger than R and this has translated into greater functionality as a programming language.\nI started using S in 1984 and moved onto R when it was first released. It’s hard to move from 34 years of experience of R to no prior experience with Python. Here are a few impressions that may help other R users who start learning Python:\n1.Base R is quite functional without loading any packages. In Python, you will always need to load some packages even to do some basic computations. You will probably need to load numpy, scipy, pandas, statsmodels, matplotlib just to get something similar to the base R environment. 2. Python is very fussy about namespaces. You will find yourself have to prefix every loaded function. For example, you cannot write log(x) — you’ll need to write np.log(x) indicating that log comes from the numpy package. I understand the reason for this but this alone makes Python code longer than R code. 3. Python array indices start from zero. Again, I know why this is but it’s something the R user has to continually adjust to. 4. matplotlib is the Python equivalent of the R base plotting functionality. It does more than R but a range of options is daunting for the new user. I had a better time with seaborn which is more like ggplot is producing attractive plots. (There’s a partial translation of ggplot into Python but I preferred seaborn). 5. statsmodels provides the linear modelling functionality found in R but you will find some differences that will trip you up. In particular, no intercept term is included by default and the handling of saturated models is different, as the Moore-Penrose inverse is used rather than dropping some offending columns as R does. The output from the linear model is far too verbose for my tastes. Of course, you can work around all these issues. 6. Python uses pipes very commonly. It helps if you have already started using these in R via the %\u0026gt;% operator to get you into that frame of mind.\nNew users inevitably encounter some frustrations but did find Python enjoyable. I have nightmares about having to do Statistics in Excel but Python world is a pleasant land even if it is still a bit unfamiliar to me.\n","date":1524787200,"expirydate":-62135596800,"kind":"page","lang":"en","lastmod":1524787200,"objectID":"e4230bc9aba8dace05be5d0c1fb99fd6","permalink":"https://julianfaraway.github.io/post/linear-models-with-r-translated-to-python/","publishdate":"2018-04-27T00:00:00Z","relpermalink":"/post/linear-models-with-r-translated-to-python/","section":"post","summary":"I have translated the R code in Linear Models with R into Python. The code is available as Jupyter Notebooks.\nI was able to translate most of the content into Python.","tags":[],"title":" Linear Models with R translated to Python","type":"post"},{"authors":[],"categories":[],"content":"More data is always better? No - not always. Nicole Augustin and I have published a paper entitled When Small Data beats Big Data. The main points are:\nQuality beats quantity. A high quality small dataset is often more informative than a biased larger dataset. Performance is a tradeoff between bias and variance. As the sample size increases, the variance decreases but the bias remains. You don\u0026rsquo;t need a huge amount of data to achieve an acceptably small variance so that small dataset with no bias due to careful sampling or a controlled experiment will beat the big garbage dump of a dataset. David beats Goliath with a well-aimed shot. Cost. There is no free lunch. More data costs money - what did you think those power studies were for? But it\u0026rsquo;s not just the acquisition costs of data. Some procedures are computationally expensive and the cost increases at a faster than linear rate with data size. If you need your results now, you might do better with less data. Other costs of data are not financial. People value privacy - we should assign a cost to invading that privacy. We should avoid using more data than necessary to protect privacy. Inference works better on small data. Statisticians have spent years developing methods for inference on relatively small datasets. Unfortunately, most of these methods don\u0026rsquo;t work well with big data because the inference becomes unbelievably sharp. The reason for this is that most statistical methods do not allow for model uncertainty or unknown sampling biases. Machine learners do no better as their methods often fail to tackle the uncertainty problem at all. Until we learn how to express uncertainty in big data models, we might be better off sticking with small data. Aggregation. Sometimes we have the option of reducing a big individual level dataset to a smaller grouped dataset. Information may be lost by this aggregation but sometimes it can be beneficial. It reduces variation, needs simpler models and reduces privacy concerns. Teaching. Students now need to learn about big and small data methods. But where to start? Small data is easier to work with. It\u0026rsquo;s much simpler for students to understand both the principles and details of the computation without the technical overhead of big data. Students will need to learn big data methods sooner rather than later but it\u0026rsquo;s best to come into this with a good understanding of the ideas of uncertainty. ","date":1522281600,"expirydate":-62135596800,"kind":"page","lang":"en","lastmod":1608214545,"objectID":"8af70e9999c72e71ccd7bc16a3b9dc77","permalink":"https://julianfaraway.github.io/post/when-small-data-beats-big-data/","publishdate":"2018-03-29T00:00:00Z","relpermalink":"/post/when-small-data-beats-big-data/","section":"post","summary":"More data is always better? No - not always. Nicole Augustin and I have published a paper entitled When Small Data beats Big Data. The main points are:\nQuality beats quantity.","tags":[],"title":"When small data beats big data","type":"post"},{"authors":["Julian Faraway","Nicole Augustin"],"categories":null,"content":"","date":1522281600,"expirydate":-62135596800,"kind":"page","lang":"en","lastmod":1522281600,"objectID":"94ab182654f36f425cf3f7cbfd03b2c2","permalink":"https://julianfaraway.github.io/publication/smalldata/","publishdate":"2018-05-01T00:00:00Z","relpermalink":"/publication/smalldata/","section":"publication","summary":"Small data is sometimes preferable to big data. A high quality small sample can produce superior inferences to a low quality large sample. Data has acquisition, computation and privacy costs which require costs to be balanced against benefits. Statistical inference works well on small data but not so well on large data. Sometimes aggregation into small datasets is better than large individual-level data. Small data is a better starting point for teaching of Statistics.","tags":null,"title":"When small data beats big data","type":"publication"},{"authors":[],"categories":[],"content":"There’s always been statistics in sports. People have kept records of achievements such as the most goals in a season or home runs in a career for many years. These statistics add extra flavour to the mere winning and losing of games and championships. I’ve no complaint about such descriptive statistics. My objection is to the use of more advanced statistical methods to improve the chances of a team to win games. The book, and later the movie, Moneyball, was about recruiting underrated players and changing the strategy of a baseball team to win more games. The underlying methods are statistical and have been very successful. The ideas have been developed and spread throughout the sporting world. Many regard this as a success story for statistics but I beg to differ.\nI understand the attraction. As a graduate student in Berkeley, I came across Bill James’ Baseball Abstract and was fascinated. James was not a professional statistician but he was great at assembling the right data to answer a question and backing up his claims with sensible statistical summaries. This was a model for how applied statistics should be done. Nonetheless, I realised that Baseball would not take much notice of a nerdy guy and so it proved for many years until Moneyball. So I’m not one of those people who object to sports sessions at Statistics conferences because sports is a trivial matter (as they feel) that does not deserve such attention at a serious conference.\nWe all like to feel that our work is improving the world in some way such as improving medical procedures or building more reliable products. American football coach, Vince Lombardi said “Winning isn’t everything, it’s the only thing.” so you might think the purpose of professional sports is winning. If statisticians can help teams win, surely that’s a good thing? But for every winner, there is a loser. The statistician can only help one team win at the expense of other teams losing. It’s a zero-sum game.\nThe true purpose of professional sports is not winning but entertainment. Lombardi had it all wrong. We watch sports because we find it enjoyable. The winning and losing are all part of the enjoyment. Do statisticians improve the enjoyment of sports? At best the answer is neutral and there is some evidence to suggest that they make it less enjoyable. In baseball, statisticians have discovered that players who foul off a lot of balls improve the chances of winning while stealing bases does not. Yet watching balls being fouled off is boring while base stealing is exciting. In football, statistical advice has sometimes led to boring, park-the-bus, defensive play.\nIt’s probably too late to turn back the clock as sports teams will not forgo the chance to gain an advantage. Nevertheless, statisticians should realise that, while they may derive some satisfaction and employment in applying statistics to sports, the overall effect of statistics on the professional sporting world has been negative.\n","date":1510876800,"expirydate":-62135596800,"kind":"page","lang":"en","lastmod":1510876800,"objectID":"73288be2a9044d8f0f5b1ba7d39bc792","permalink":"https://julianfaraway.github.io/post/statisticians-do-not-improve-sports/","publishdate":"2017-11-17T00:00:00Z","relpermalink":"/post/statisticians-do-not-improve-sports/","section":"post","summary":"There’s always been statistics in sports. People have kept records of achievements such as the most goals in a season or home runs in a career for many years. These statistics add extra flavour to the mere winning and losing of games and championships.","tags":[],"title":"Statisticians do not improve sports","type":"post"},{"authors":[],"categories":[],"content":"Almost every student has a smart phone so it makes sense to format lecture notes so that they can be read on these small screen devices. But this can be difficult to achieve if you use LaTeX to produce your lecture notes or other mathematical/statistical handouts. We also want to maintain a full size version so the smaller version needs to be produced with minimal changes. Here are some tips which require increasing effort to implement\nUse A6 size paper by opening your LaTeX document with: \\documentclass[a6paper]{article} This is one quarter the size of A4 so it will shrink the page size greatly. I use the default 10pt font since students tend to have good eyesight. I can just about read this if I hold my phone in landscape orientation.\nUse the geometry package to specify minimal margins: \\usepackage[margin=1mm]{geometry} Because who needs margins if you are not printing this out.\nI use the graphicx package for including graphics. When including a plot or diagram use something like this: \\includegraphics*[width=0.75\\textwidth]{resfit.pdf} The plot will take up 75% of the width of the page which works for both the small and the large screen versions.\nSome mathematical expressions can be quite long and will exceed the page width particularly on the small screen version. This is not a problem with text since LaTeX knows how to set the line breaks. But it’s much harder to do this with mathematical expressions. This is where the breqn package is very useful. At a minimum you can easily replace all your equation environments with dmath. This will get you automated line breaking in your equations. The breqn package has a lot more functionality if you want to make more of an effort.\nIt would be nice if LaTeX could produce documents that could dynamically reflow depending on screen size like the *epub\u0026amp; format used on e-readers. But that’s unlikely to happen so a version formatted for the small screen is the next best thing.\n","date":1474416000,"expirydate":-62135596800,"kind":"page","lang":"en","lastmod":1474416000,"objectID":"6ed06703e8ac21486a328376867951d0","permalink":"https://julianfaraway.github.io/post/formatting-latex-for-a-small-screen/","publishdate":"2016-09-21T00:00:00Z","relpermalink":"/post/formatting-latex-for-a-small-screen/","section":"post","summary":"Almost every student has a smart phone so it makes sense to format lecture notes so that they can be read on these small screen devices. But this can be difficult to achieve if you use LaTeX to produce your lecture notes or other mathematical/statistical handouts.","tags":[],"title":"Formatting LaTeX for a small screen","type":"post"},{"authors":[],"categories":[],"content":"Today, the Guardian published its University Guide 2017. Let’s take a closer look at the components of these league tables. I don’t mean to pick on the Guardian table as the other tables have similar characteristics.\nThree of the measures are based on student satisfaction and are drawn from the National Student Survey. When the NSS started out, it received fairly honest responses and had some value in provoking genuine improvements in education. But its value has deteriorated over time as Universities and students have reacted to it. Most students realise that the future value of their degree depends on the esteem in which their university is held. It is rational to rate your own university highly even if you don’t really feel that way. Furthermore, students find it difficult to rate their experience since most have only been to one university. It’s like rating the only restaurant you’ve ever eaten at. The Guardian makes things worse by using three measures from the NSS in its rankings.\nStudent to staff ratio is a slippery statistic. Many academics have divided responsibilities between teaching and research. It’s difficult to measure how much teaching they do and how much they interact with students. Class sizes can vary a lot according to type of programme and year in the degree – it’s not like primary education. Spend per student is another problematic measure. Expenditure on facilities can vary substantially from year to year and unpicking university budgets is difficult.\nAverage entry tariff is a solid measure and reflects student preferences. If you reorder the Guardian table on this measure alone, you’ll get a ranking that is closer to something knowledgeable raters would construct. This measure is sensitive to the selection of courses offered by the university since grades vary among A-level subjects.\nValue added scores are highly dubious. It’s a measure of the difference between degree output and A-level input. A-levels are national tests and are a reasonable measure of entry ability. Degree qualifications are not comparable across Universities. A 2:1 at a top university is not the same as a 2:1 from a lower level university. If you compare the exams in Mathematics taken in top universities with those given by lower level universities, you will see huge differences in the difficulty and content. A student obtaining a 2:2 in Mathematics at a top university will likely know far more Maths than a student with a first from a post 92 university. This means it is foolish to take the proportion of good degrees (1 and 2:1) as a measure of output performance.\nThe final component is the percentage holding career jobs after six months. This is a useful statistic but is hard to measure accurately. This will also be affected by the mixture of courses offered by the university.\nAll these measures are then combined into a single “Guardian score”. There is no one right way to do this. If you consider the set of all convex combinations of the measures, you would generate a huge number of possible rankings, all them just as valid (or invalid) as the Guardian score. It’s a cake baked from mostly dodgy ingredients using an arbitrary recipe.\nWe might laugh this off as a bit of harmless fun. Unfortunately, some prospective students and members of the public take it seriously. Universities boast of their performance in press releases and on their websites. University administrators set their policies to increase their league table standing. Some of these policies are harmful to education and rarely are they beneficial. Meanwhile, the league tables are not actually useful to a 17 year old deciding on a degree course. The choice is constrained by expected A-level grades, course and location preferences. The statistics in the tables fluctuate from year to year and are an unreliable predictor of the individual student experience.\n","date":1463961600,"expirydate":-62135596800,"kind":"page","lang":"en","lastmod":1463961600,"objectID":"b4806b5243b901942e7365e8394fc208","permalink":"https://julianfaraway.github.io/post/what-s-wrong-with-university-league-tables/","publishdate":"2016-05-23T00:00:00Z","relpermalink":"/post/what-s-wrong-with-university-league-tables/","section":"post","summary":"Today, the Guardian published its University Guide 2017. Let’s take a closer look at the components of these league tables. I don’t mean to pick on the Guardian table as the other tables have similar characteristics.","tags":[],"title":"What’s wrong with University League tables","type":"post"},{"authors":[],"categories":[],"content":"Suppose you have some data and you want to build a model to predict future observations. Data splitting means dividing your data into two, not necessarily equal, parts. One part is used to build the model and the other half to evaluate it in some way. There are two related, but distinct, reasons why people do this. It’s well known that if you use the same data to both build the model and test the performance of that model, you’ll be overoptimistic about how well your model will do in predicting new data. Analysts have various tricks to avoid overconfidence, such as crossvalidation, but these are not perfect. Furthermore, if you need to prove to someone else how well you’ve done, they will be sceptical of such tricks. The gold standard is to reserve part of your data as a test set and build and fit your model on the remaining training set part of the data. You use this model to predict the observations in the test set. Because the test set has been held back, it’s (almost) like having fresh data. The performance on this test set will be a realistic assessment of how the model will perform on future data. But you lost something in the data splitting – the training data is smaller than the full data so the model you select and fit won’t be as good. If you don’t need to prove how good your model is, this form of data splitting is a bad idea. It’s as if a customer orders a new car and asks that the seller drive it around for 10K miles to prove there’s nothing wrong with it. The customer will receive the assurance that the car is not a lemon but it won’t be as good as getting a brand new car in perfect condition.\nBy the way, if you find your model doesn’t do as well as you’d hoped on the test set, you might be tempted to go back and change the model. But the performance on this new model cannot be judged cleanly with the test set because you’ve borrowed some information from the test set to help build the model. You only get one shot using the test set.\nThere’s a second reason why you might use data splitting. The typical model building process involves choosing from a wide range of potential models. Usually there is substantial model uncertainty about the choice but you take your best pick. You then use the data to estimate the parameters of your chosen model. Statistical methods are good at assessing the parametric uncertainty in your estimates, but don’t reflect the model uncertainty at all. This a big reason why statistical models tend to be overconfident about future predictions. That’s where the data splitting can help. You use the the first part of your data to build the model and the second part to estimate the parameters of your chosen model. This results in more realistic estimates of the uncertainty in predictions. Again you pay a price for the data splitting. You use less data to choose the model and less data to fit the model. Your point predictions will not be as good as if you used the full data for everything. But your statements of the uncertainty in your predictions will be better (and probably wider).\nSo judging the value of data splitting in this context means we have to attach some value to uncertainty assessment as well as point prediction. The method of scoring is a good way to do this. All this and more is discussed in my paper Does data splitting improve prediction? and on ArXiv. Although it depends on the circumstances, I show that this form of data splitting can improve prediction.\n","date":1460073600,"expirydate":-62135596800,"kind":"page","lang":"en","lastmod":1460073600,"objectID":"95054bf58c3b7043add9d0d5887379cc","permalink":"https://julianfaraway.github.io/post/is-data-splitting-good-for-you/","publishdate":"2016-04-08T00:00:00Z","relpermalink":"/post/is-data-splitting-good-for-you/","section":"post","summary":"Suppose you have some data and you want to build a model to predict future observations. Data splitting means dividing your data into two, not necessarily equal, parts. One part is used to build the model and the other half to evaluate it in some way.","tags":[],"title":"Is data splitting good for you?","type":"post"},{"authors":[],"categories":[],"content":"In What’s wrong with simultaneous confidence bands, I discussed the deficiencies of standard confidence bands, but how can we do better? Suppose we have a Gaussian process for the curve with some mean and a covariance kernel:\n$$ k(x,x^\\prime) = \\sigma^2_f \\exp \\left( - {1 \\over 2l^2} (x-x^\\prime)^2 \\right) + \\sigma^2_n \\delta(x - x^\\prime) $$\nwhere δ is a delta function. The three hyper parameters, $\\sigma_f, \\sigma_n$ and $l$ control the appearance of the curve. We can put priors on these parameters along with a prior on the mean function and calculate the posterior using MCMC methods. The most important parameter $l$ controls the smoothness of the curve. We can view the uncertainty in this smoothness by computing a 95% credible interval for this parameter and plot the two curves corresponding to the endpoints of this interval. The other parameters are set at the posterior means. An example of these bands can be seen in the figure:\nPlot of smoothness bands The solid line corresponds to the lower end of 95% interval for the smoothing parameter $l$ giving a rougher fit while the dashed line is the upper end of the interval which is a smoother fit. We are fairly sure that the correct amount of smoothness lies between these two limits.\nNotice this makes a big difference to how many peaks and valleys we see in the function. The uncertainty about these features is more interesting than any uncertainty about the vertical position of the curve expressed by the traditional bands.\nYou can read all the details in my article: Confidence bands for smoothness in nonparametric regression and also view the R code.\n","date":1458086400,"expirydate":-62135596800,"kind":"page","lang":"en","lastmod":1608215995,"objectID":"b95595f012a663ae7cdb7a3cccbd0884","permalink":"https://julianfaraway.github.io/post/confidence-bands-for-smoothness/","publishdate":"2016-03-16T00:00:00Z","relpermalink":"/post/confidence-bands-for-smoothness/","section":"post","summary":"In What’s wrong with simultaneous confidence bands, I discussed the deficiencies of standard confidence bands, but how can we do better? Suppose we have a Gaussian process for the curve with some mean and a covariance kernel:","tags":[],"title":"Confidence Bands for Smoothness","type":"post"},{"authors":[],"categories":[],"content":"Sky surveys provide a wide-angle view of the universe. Instead of focusing on a single star or galaxy, they provide a snapshot of a wider portion of the sky. Their primary purpose is to search for asteroids, so they take a sequence of images over a short timespan: minutes or hours. By seeing what has changed, the asteroid can be detected. But meanwhile, much further away, other things are changing. Most objects in the night sky do not change much on human timescales, but some of the most interesting stars and galaxies are performing for us. As the survey returns to the same location from time to time, we can see those objects that have changed. But how do we detect and categorise these objects? Here is a plot of four such objects:\nPlot of light curves The first of the four is an example of an active galactic nucleus. The plot shows how the brightness of the object varies over time. The second plot shows a supernova – most of the time we see nothing because the galaxy is below the detection limit but briefly we see a spark of light for a few weeks. The third object is a flare – it’s mostly quiet but burst into life from time to time. The fourth object is like our sun – it varies a little but enough to be interesting (fortunately for us!).\nThese are examples of light curves. As technology improves, increasingly large numbers of these curves will be recorded. We need to quickly identify and categorise the interesting ones so that they can be rapidly followed up (before the show is over!). In Modeling lightcurves for improved classification of astronomical objects, we describe how this can be done. You can also read about it on ArXiv and examine the software and data.\n","date":1457827200,"expirydate":-62135596800,"kind":"page","lang":"en","lastmod":1457827200,"objectID":"705fb105680fb40d773416cbf5d7b947","permalink":"https://julianfaraway.github.io/post/modeling-light-curves-for-improved-classification-of-astronomical-objects/","publishdate":"2016-03-13T00:00:00Z","relpermalink":"/post/modeling-light-curves-for-improved-classification-of-astronomical-objects/","section":"post","summary":"Sky surveys provide a wide-angle view of the universe. Instead of focusing on a single star or galaxy, they provide a snapshot of a wider portion of the sky. Their primary purpose is to search for asteroids, so they take a sequence of images over a short timespan: minutes or hours.","tags":[],"title":"Modeling Light Curves for Improved Classification of Astronomical Objects","type":"post"},{"authors":[],"categories":[],"content":"Here are some 95% confidence bands for a fitted curve shown on the same data. The first uses spline smoothing from the mgcv package while the second uses a loess smoother from the ggplot2 package.\nPlot of smoothness bands using mgcv Plot of smoothness bands using loess If all you want is a graphical expression of the uncertainty in these two estimates, these are just fine. But suppose you want to check whether some proposed function fits entirely within the bands, then you will need to do more work. The bands above are pointwise meaning that the confidence statement is true at any given point but not true for the entire curve. For that, you will need simultaneous confidence bands. These are more work to produce and involve all sorts of intricate calculations. Hundreds of papers have been written on the topic because of the fascinating theoretical challenges it raises. I\u0026rsquo;m responsible for a few of these papers myself.\nBut are these bands really useful? Properly constructed, they may tell us there\u0026rsquo;s a 95% chance that the bands capture the true curve. But what is the value in that? There will be some users who are interested in replacing the smooth fit with some particular parametric form, say a quadratic. Such users would be better off embedding their search within a family of increasingly complex alternatives and choosing accordingly. SCBs would not be an efficient strategy.\nIn SCB papers that have a data example, the usual motivation is that the user is interested in whether some particular feature exists, say a secondary maximum, for example. But users looking for particular features are better off with a method designed to look for those features, such as SiZer.\nThe greatest uncertainty is demonstrated by comparing the two figures above. How much smoothing should be applied? This makes a crucial difference to our interpretation. In the typical SCB construction, the amount of smoothing is chosen by some algorithm and the bands only reflect the uncertainty in the amplitude of the curves.\nWe need bands that tell us about the uncertainty in the smoothing. I will explain how to do this in the next blog post.\n","date":1457654400,"expirydate":-62135596800,"kind":"page","lang":"en","lastmod":1608217150,"objectID":"38c11c25b015d8509505e9b393042fe8","permalink":"https://julianfaraway.github.io/post/what-s-wrong-with-simultaneous-confidence-bands/","publishdate":"2016-03-11T00:00:00Z","relpermalink":"/post/what-s-wrong-with-simultaneous-confidence-bands/","section":"post","summary":"Here are some 95% confidence bands for a fitted curve shown on the same data. The first uses spline smoothing from the mgcv package while the second uses a loess smoother from the ggplot2 package.","tags":[],"title":"What's wrong with simultaneous confidence bands","type":"post"},{"authors":["Julian Faraway"],"categories":null,"content":"","date":1452384000,"expirydate":-62135596800,"kind":"page","lang":"en","lastmod":1452384000,"objectID":"81142c62ff2b22d382b1a3b8a018029f","permalink":"https://julianfaraway.github.io/publication/confidence-bands/","publishdate":"2016-01-10T00:00:00Z","relpermalink":"/publication/confidence-bands/","section":"publication","summary":"The choice of the smoothing parameter in nonparametric regression is critical to the form of the estimated curve and any inference that follows. Many methods are available that will generate a single choice for this parameter. Here, we argue that the considerable uncertainty in this choice should be explicitly represented. The construction of standard simultaneous confidence bands in nonparametric regression often requires difficult mathematical arguments. We question their practical utility, presenting several deficiencies. We propose a new kind of confidence band that reflects the uncertainty regarding the smoothness of the estimate.","tags":null,"title":"Confidence bands for smoothness in nonparametric regression","type":"publication"},{"authors":[],"categories":[],"content":"While looking at tables of contents for journals and CVs for job applications, it struck me that there are a lot of multi-author papers now. Just to check that I was not imagining it, I downloaded the tables of contents for JASA and JRSS-B for the years 2014, 2004, 1994, 1984 and 1974. I removed the comments, book reviews and corrections to focus just on the research articles. I counted the number of authors for each paper and here’s what I found:\nMean number of authors per article:\nJournal 1974 1984 1994 2004 2014 JASA 1.48 1.61 1.89 2.50 2.97 JRSS 1.29 1.51 1.70 2.45 2.65 and\nFraction of single author articles:\nJournal 1974 1984 1994 2004 2014 JASA 0.56 0.50 0.35 0.19 0.06 JRSS 0.71 0.57 0.41 0.12 0.03 I’d like to have done more but Web of Science doesn’t make it easy to get the data. Nevertheless, the trend is very clear. Browse through a few journals, old and new and you’ll see the same pattern. You can see similar trends in other fields. The average number of authors per paper has been increasing significantly over time. Most dramatically, the number of single author papers has gone from a majority down to single digit percentages.\nNow there are several good reasons not to go it alone. If you write a genuine applications paper, you will be collaborating with a scientist from another field who would be a natural co-author. But most of the articles in these two journals are methodological with Statistical authors. In any case this doesn’t explain the trend. It’s also possible that your co-authors bring different skills to the table, theoretical or computational. It’s also nice to have someone to discuss the paper. You correct each other’s errors and misconceptions. You learn from each other. The social aspect of the collaboration can make you more productive. But that’s also been true in the past. You can point to the greater ease of long distance collaborations using the internet but I suspect there is a stronger cause.\nThe academic world is increasingly driven by metrics. You will get far more credit for writing four articles, each with four authors than for producing one single author paper. So there’s a huge incentive to collaborate. This is good when the co-authors have clearly distinct skills but often they are just other statisticians, much like you.\nBut something has been lost. Most good papers have one main idea and only one person can have that idea. Sure, other people can help refine that idea and turn it into a paper but somewhere in that list of multiple authors there is that one person who had the idea. Truly creative and innovative ideas start out sounding a bit crazy, ill-formed and speculative. If you discuss the idea with your peers, they will back away since it won’t sound like something that will reliably result in a publishable paper. Don’t even think of submitting a grant proposal. So you work on the idea and turn it into flesh. By that time, you don’t need co-authors. Working as a group is a reliable way to get work published but for truly creative work, you need to go it alone.\n","date":1449619200,"expirydate":-62135596800,"kind":"page","lang":"en","lastmod":1449619200,"objectID":"87dc8257b7ce74f15f65747403f888d6","permalink":"https://julianfaraway.github.io/post/statistics-authors-single-no-more/","publishdate":"2015-12-09T00:00:00Z","relpermalink":"/post/statistics-authors-single-no-more/","section":"post","summary":"While looking at tables of contents for journals and CVs for job applications, it struck me that there are a lot of multi-author papers now. Just to check that I was not imagining it, I downloaded the tables of contents for JASA and JRSS-B for the years 2014, 2004, 1994, 1984 and 1974.","tags":[],"title":"Statistics authors single no more","type":"post"},{"authors":null,"categories":["R"],"content":" Consider a regression dataset with a response and several predictors. You want a single plot showing the response plotted against each of the predictors. You could use the pairs() but that also shows plots between the predictors. If there are more than a few predictors, there are too many plots to see any one of them clearly. Here’s a simple solution:\nHere’s an example dataset:\nhead(swiss) Fertility Agriculture Examination Education Catholic Courtelary 80.2 17.0 15 12 9.96 Delemont 83.1 45.1 6 9 84.84 Franches-Mnt 92.5 39.7 5 5 93.40 Moutier 85.8 36.5 12 7 33.77 Neuveville 76.9 43.5 17 15 5.16 Porrentruy 76.1 35.3 9 7 90.57 Infant.Mortality Courtelary 22.2 Delemont 22.2 Franches-Mnt 20.2 Moutier 20.3 Neuveville 20.6 Porrentruy 26.6 Now reorganise the data using the tidyr package so that there is one (x,y) pair on each line:\nlibrary(tidyr) rdf \u0026lt;- gather(swiss, variable, value, -Fertility) head(rdf) Fertility variable value 1 80.2 Agriculture 17.0 2 83.1 Agriculture 45.1 3 92.5 Agriculture 39.7 4 85.8 Agriculture 36.5 5 76.9 Agriculture 43.5 6 76.1 Agriculture 35.3 Use ggplot2 to plot the response against each of the predictors (which are on different scales so we need to allow for that)\nlibrary(ggplot2) ggplot(rdf, aes(x=value,y=Fertility)) + geom_point() + facet_wrap(~ variable, scale=\u0026quot;free_x\u0026quot;) We can elaborate as needed.\n","date":1444348800,"expirydate":-62135596800,"kind":"page","lang":"en","lastmod":1444348800,"objectID":"10065deaa3098b0da91b78b48d0efc71","permalink":"https://julianfaraway.github.io/post/2015-07-23-r-rmarkdown/","publishdate":"2015-10-09T00:00:00Z","relpermalink":"/post/2015-07-23-r-rmarkdown/","section":"post","summary":"Consider a regression dataset with a response and several predictors. You want a single plot showing the response plotted against each of the predictors. You could use the pairs() but that also shows plots between the predictors.","tags":["R Markdown","plot","regression"],"title":"Plotting Regression Datasets","type":"post"},{"authors":[],"categories":[],"content":"There are a very large number of books, articles and university courses with “Generalised Linear Models” (GLM) in the title. I even wrote such a book myself. But is the GLM paradigm really that useful?\nThe idea is that we can develop a general theory of estimation, inference and diagnostics that will apply to a wide class of models. We will avoid duplication of effort and synergies will arise across this class of models. The idea goes back to a 1972 paper in JRSS-B by Nelder and Wedderburn.\nBut what response distributions belong to the GLM family? The Gaussian (or normal) distribution is the most used in practice. But this is simply the linear model for which much more general and powerful results exist. The Gaussian gains nothing from the GLM perspective.\nNext we have the binomial and Poisson, which do benefit from GLM membership. But beyond that, the family members become progressively more exotic. The gamma GLM is not commonly used because a linear model with a transformed response will often suffice. Venturing further into the outback, you may find an inverse Gaussian or Tweedie GLM but these are truly rare birds. There are more interesting distributions such as the negative binomial and beta but they are excluded from the club as they don’t belong to the “exponential family” of distributions.\nSo there are only two important members of the GLM family: the binomial and the Poisson. Even here we must be careful because large chunks of the theory and practice for these models is specific to one or the other. The GLM paradigm does tell us one way to estimate and do inference for these models. But there other ways to do these things.\nStatisticians love widely applicable theories – but perhaps a little too much. GLM is a nice, useful theory but the paradigm has become too dominant in the way people learn or are taught about these kind of models.\n","date":1439942400,"expirydate":-62135596800,"kind":"page","lang":"en","lastmod":1439942400,"objectID":"7a6916cbe4a8e9c934fd78b165d0920a","permalink":"https://julianfaraway.github.io/post/glm/","publishdate":"2015-08-19T00:00:00Z","relpermalink":"/post/glm/","section":"post","summary":"There are a very large number of books, articles and university courses with “Generalised Linear Models” (GLM) in the title. I even wrote such a book myself. But is the GLM paradigm really that useful?","tags":[],"title":"GLM - An overused paradigm","type":"post"},{"authors":["Julian Faraway"],"categories":null,"content":"","date":1414540800,"expirydate":-62135596800,"kind":"page","lang":"en","lastmod":1414540800,"objectID":"7e482fa638748f3e9146b4b88ca6ddd5","permalink":"https://julianfaraway.github.io/publication/data-splitting/","publishdate":"2014-10-29T00:00:00Z","relpermalink":"/publication/data-splitting/","section":"publication","summary":"Data splitting divides data into two parts. One part is reserved for model selection. In some applications, the second part is used for model validation but we use this part for estimating the parameters of the chosen model. We focus on the problem of constructing reliable predictive distributions for future observed values. We judge the predictive performance using log scoring. We compare the full data strategy with the data splitting strategy for prediction. We show how the full data score can be decomposed into model selection, parameter estimation and data reuse costs. Data splitting is preferred when data reuse costs are high. We investigate the relative performance of the strategies in four simulation scenarios. We introduce a hybrid estimator that uses one part for model selection but both parts for estimation. We argue that a split data analysis is prefered to a full data analysis for prediction with some exceptions.","tags":null,"title":"Does data splitting improve prediction?","type":"publication"},{"authors":null,"categories":null,"content":"Formerly, I worked at the Human Motion Simulation at the University of Michigan. You can read more about my work here\n","date":1166659200,"expirydate":-62135596800,"kind":"page","lang":"en","lastmod":1166659200,"objectID":"e6b139a595a1fb0719f9f3b3cbd1a05e","permalink":"https://julianfaraway.github.io/project/ergonomics/","publishdate":"2006-12-21T00:00:00Z","relpermalink":"/project/ergonomics/","section":"project","summary":"Human motion modelling with automotive applications","tags":["research"],"title":"Ergonomics","type":"project"}]