Directors' Blog
Bad Science and Beyond
“I just don’t care!” the doctor said, in response to a query on a personal health-related issue during a recent New Zealand television interview.
I certainly had never heard a doctor speak in this way – and it made me curious to hear more in his live talk.
I was not disappointed. The thought-provoking comments kept coming: “The right way to build public trust is to earn public trust, and to share data” 
and, “The paradigm of medicine has somewhat shifted”.
In his unique and energetic style, Dr. Ben Goldacre got his message across to the audience at the Mercury Theatre in Auckland this September. A British physician, researcher, columnist, and author, Dr. Goldacre has made it his mission to tackle “bad science”, whether it is used by drug companies, politicians, journalists, or researchers.
Dr. Goldacre explained the misrepresentation of the research life cycle (objective – data collection – data analysis – publication – evaluation), showing the audience the easiest way to mispresent science and the shortcomings of medicine.
Using statistical data taken from newspapers, advertisements, and the research reports, the audience learned how so-called “in-depth scientific research” can be used as a clever marketing tool and how defined research objectives can often by driven by profit.
Photo credit: Dr Michelle Dickinson
One telling example of misrepresented data is the sudden drop in the number of tonsillectomies carried out on children in Hornsey, North London. There was a big discrepancy before and after 1929 – that is, from a few hundred cases down to almost nothing. After some investigation, it was revealed that the decline in the number of tonsil operations coincided with the retirement of one individual medical officer at a particular school, replaced by someone with a different opinion as to the merits of the treatment. Such a case reveals the power of doctors’ choices, rather than patients’ needs. It also shows that how unreliable the data will be if the full picture is not disclosed.
Sharing a more recent case, Dr. Goldacre explained the use of statins, a medicine to lower cholesterol in the blood. Many treatment options are available to lower cholesterol against a placebo, but these have not been tested against one another to determine “real world effects”, including death. Dr. Goldacre and his team approached the UK National Health Service (NHS) to collect patient information. If patients agreed, doctors would be able to randomize the treatment options, ultimately finding the optimal treatment for considerably less resource compared to the traditional “door knocking” data collection method. Nevertheless, this was thwarted by opposition from ethics groups arguing that patients should have a choice.
Dr. Goldacre’s presentation led me to wonder; what role does the general public play in formulating bad science? Surely it is not just the domain of motivated organisations, unreliable researchers, and sensationalist media. Bad science can, and often is, disseminated by all walks of life.
How can we stop its spread? Transparency may be the answer – including that of research objectives, processes, and publication. If research objectives are set for the benefit of all, or purely for the improvement of a company’s bottom line, the public should know – and in a language they can understand. Let consumers make up their own minds.
Unfortunately, providing scientific data to the wider public and expecting people to reach their own conclusions may not be sufficient. Effective communication is also a critical element in combatting bad science. Nowhere is this more eloquently stated than in Professor Shaun Hendy’s timely book, Silencing Science: “The job of the scientist is not just to deliver the facts, but also to engage democratically to assist the community to weigh the full breadth of evidence” (p96).
Indeed, scientists and research providers should communicate well, working together in the best interests of the public.
As a consumer and a citizen in a world of information overload, it can be easy to be misled by a well-packaged data snapshot. We need to embrace a reliable and complete picture, and in terms we can understand. This will allow us to make our own choices in areas as broad as health, education, career, life-style, and more.
As a student of science and maths, I am beginning to grasp the moral imperative of the scientific community. Scientists should not only be answerable to their fund providers, but to everyone. A good start would be the publication of research findings representing the whole truth.
About
14-year-old Tristan Pang is a maths and physics major at the University of Auckland. He is also the creator of Tristan’s Learning Hub, producer and broadcaster of Youth Voices, founder and webmaster of several community websites, frequent speaker at schools, organizations and conferences, and tutors students from primary school level through to university. He aspires to make a difference in the world.
Science for Policy: Part I
A good deal of the research we do at Te Pūnaha Matatini is intended to inform government policy. But it is one thing to do the research, and quite another to have that research influence policy. This is why there is a growing interest in the relationship between research and policy, although there are still many different points of view on what this relationship should, let alone does, take. Over the next month or so we are going to post a series of blogs that discuss some of the issues that face researchers who wish to influence policymakers and policymakers who wish to use research.
In this first post, I am going to review aspects of this issue that are touched on in my recent book, Silencing Science. There I discuss the reasons why so few scientists seem to be prepared to engage with the public on subjects that are politically contested (tl;dr? Try this article from the Education Review). There are lots of motivations for avoiding contentious debates in public, but one concern that scientists have is the risk of damaging their relationship with policymakers, with the consequent implications for funding and their job. Understanding this relationship is important if we want to improve the use of research in policy.
The model I used to analyse this relationship in Silencing Science was borrowed from Roger Pielke, based on the analysis in his book The Honest Broker. He identifies four modes in which scientists can legitimately engage with policymakers: the pure scientist, the science arbiter, the issues advocate, and the honest broker of policy alternatives. As I wrote:
“The first two modes operate when a scientist provides advice on issues with policy options around which there is political consensus. The pure scientist simply summarises the state of knowledge in a particular field without reference to policy options. If a scientist is asked by a policy-maker to weigh in on the evidence for or against the effectiveness of a specific policy option, they adopt the role of science arbiter. In both cases, the scientist can claim to be sticking to the science, and can put themselves at arms length from the politics of the day.”
I would argue that these two modes dominate the approach that New Zealand scientists take to engaging with government. These are the silent scientists; they may engage behind the scenes with policy-makers, but they generally don’t make an effort to inform the public other than through very passive channels (e.g. see the Royal Society of New Zealand’s report on the water fluoridation). Pielke argues that these modes are appropriate when the policy implications are not politically divisive, but when policies have serious political ramifications, Pielke says that a different approach is needed.
From Silencing Science again:
“The situation is more complex for the science advisor when providing advice on policies that are politically divisive. In this case, Pielke argues that the roles of the pure scientist or the science arbiter are poor choices. By standing back from politics, Pielke says, scientists risk becoming pawns in a contested public debate. When scientists claim they are sticking to the science on hotly contested issues, their scientific authority can be hijacked by special interests.”
The recent inquiry into the Ministry of Primary Industries’ (MPI) failure to prosecute over illegal fish discards illustrates this. The inquiry found emails from an MPI senior manager in 2014 that revealed serious concerns about the way illegal fish discards were being monitored:
“discarding is a systemic failure of the current system and something we have not been able to get on top of from day 1 of the QMS [Quota Management System]. FM [Fisheries Management] can’t quantify the tonnages involved but we suspect they are significant to the point that they are impacting on stocks.”
Yet in May 2016, prior to the release of these emails, the same senior manager was quoted in an MPI press release saying:
“Science is the bedrock of our approach to fisheries management and New Zealand invests $22.5 million each year to ensure our fisheries science is up-to-date and accurate.”
This response makes me very uncomfortable. The Ministry is using the authority of science to deflect criticism and legitimate public scrutiny of the strengths and weaknesses of its management systems.
In this type of situation, Pielke suggests that scientists are better to approach the issue as an advocate, or an honest broker. The advocate takes sides in a policy debate, openly going beyond the science to grapple with the policy implications that may stem from the science. Indeed, the fisheries story and the inquiry itself were sparked by University of Auckland researcher Glenn Simmons arguing for much stronger monitoring of discards:
“… the future sustainability, traceability and certification of fisheries will depend on how government addresses the under-reporting problems, which have long been evident and which should be a cause of concern. Unreported catches and dumping not only undermine the sustainability of fisheries, but result in a suboptimal use of fishery resources and economic waste of valuable protein.”
Simmons’ role in the debate is not something that many scientists would relish. He has been subject to criticism by the Ministry and has his work critiqued in the media by his peers – while peer critique is a crucial part of science, scientists are not always comfortable when it takes place in the public eye. Nonetheless, advocates like Simmons play a crucial role in getting issues on the policy agenda.
The trick to pulling this off, according to Pielke, is to avoid using your science to mask a hidden agenda. An advocate must be explicit about where the science ends and values take over, acknowledging that scientific evidence alone is not sufficient in itself to make a policy decision.
The fourth option is that of the honest broker. In this role, the honest broker, like the advocate, acknowledges the gap between science and policy. Rather than trying to weigh in on a particular side of a policy debate, though, the honest broker attempts to consider a range of possible policy options, perhaps even using their expertise to introduce new solutions that were not yet on the table.
The honest broker is perhaps the most difficult stance for an individual researcher to attempt. Individuals are very rarely in a position where this is practical, as it requires the synthesis of the expertise of a wide range of colleagues and a diverse set of political viewpoints. In Silencing Science, I single out the Parliamentary Commissioner for the Environment as an example of an honest broker, but the Commissioner is supported in that role by a large team and is in the position to take a significant amount of time and care in weighing in on issues. As Pielke has pointed out, honest brokers are almost never a single individual. More typically this is a role for committees or panels.
In New Zealand we have several bodies that might be in a position to take honest broking on. The Royal Society of New Zealand “produce papers, convene panels and hold events to provide expert advice to policy-makers and contribute to public debate.” Generally this advice is undertaken in the pure scientist or science arbiter mode: a recent advice paper on sugar and health, for instance, almost entirely avoids policy recommendations, focussing instead on summarising evidence linking sugar consumption and health, despite the intense debate around policy options such as sugar taxes and mandatory labelling.
The other significant body is the Network of Science Advisors chaired by Sir Peter Gluckman, the Prime Minister’s Chief Science Advisor. The terms of reference and membership of this group is not readily available to the public, so it is difficult to comment on the way they operate. We are going to be discussing this group in a later post, together with some recommendations about how we think it could be utilised more effectively.
While Pielke’s model is a useful entry point into this discussion, it does have a number of shortcomings. Over the next few weeks we’ll be discussing this further in the New Zealand context.
Shaun Hendy
Healthcare Analytics Workshop
Te Pūnaha Matatini is sponsoring a Health Analytics Workshop following the 2016 Joint NZSA+ORSNZ Conference.
What: Health Analytics Workshop following the Joint NZSA+ORSNZ Conference
When: Thursday 1 December, 2016 (full day)
Where: AUT City Campus
Registration: Workshop participants, including those not attending the conference, can register for the workshop on the conference registration page.
The aim of the workshop is to bring together practitioners and researchers in healthcare analytics. People with problems meeting people with solutions! Practitioners – please bring along your current “pain point(s)”. Researchers – please talk about your success stories with the health sector! We look forward to an exciting, productive workshop. If you have any questions please contact Principal Investigators Ilze Ziedins (i.ziedins@auckland.ac.nz) or Mike O’Sullivan (michael.osullivan@auckland.ac.nz), or Associate Investigator Cameron Walker (cameron.walker@auckland.ac.nz).
Programme
- 10-11:20 Researcher Presentations
- 11:20-11:40 Morning tea
- 11:40-1 Industry Presentations
- 1-2:30 lunch
- 2:30-4 Facilitated Networking Session
- 4-5 Drinks
Note
- If you wish to give a presentation please contact Ilze Ziedins.
- Workshop organisers will post titles of presentations closer to the event at the conference website, along with information about the facilitated networking session.
Ngā Pae o te Māramatanga summer internship
Applications are now open for student summer internships at Ngā Pae o te Māramatanga, New Zealand’s Māori Centre of Research Excellence. Projects include one at the Auckland City Council titled: “Indigenous knowledge as evidence in local government decision-making: challenges and opportunities.”
Get in quick! Applications close Monday October 24.
Find out more and apply at the Ngā Pae o te Māramatanga website.
A more social network
In the immortal words of Vanilla Ice – Stop, collaborate and listen. Collaboration is a cornerstone of modern science and with flight tickets cheaper than ever before and the internet effectively eliminating the expense of correspondence, academics and researchers are looking further afield and reaching more contemporaries across the globe. However, different institutions have different facilities and research focuses, not everyone speaks the same language, and so perhaps these researchers may be picky when it comes to who they work with. It raises the question of whether they do have a preference in collaborator based on affiliation and, if so, can this preference be measured and distilled into cold, hard data?
Of course they do, and of course it can be. More to the point, why?
Arguably the most tangible and conveniently quantifiable means in which academic collaboration manifests is in scientific papers and articles, typically with several authors from varying affiliations. A notable drawback in previous studies on research collaboration is that the measures used (such as the fractional count detailed in Nature Index) consider results for each institution, rather than individual academic, and disregard the size of each institution; as a result, smaller and younger institutions may stack up unfavourably compared to those that are more established and larger. For example, take a look at how the eight New Zealand universities compare against each other:
- The nodes representing each university are weighted by their respective output (total number of co-authored papers by academics affiliated with these universities).
- The links connecting universities to each other are weighted by the number of papers co-authored by researchers from both institutions.
- The higher the link weight, the more that the connected universities are attracted to each other.
The skewing effect that university size has on this network is pretty apparent from how Lincoln University has much fewer co-authorships with Victoria University and University of Waikato than with the rest of the network, given its relatively small output. Also of note is that the University of Auckland and AUT have a much lower link weight than one would expect for two universities across the street from each other, yet the University of Auckland and the University of Canterbury have a much stronger link despite being at opposite ends of the country.
First, to address the effect of institution output. We do this using something we call the revealed comparative preference (RCP) of an institution i for collaborating with institution j:
where Xij is the number of co-authorships between i and j, Xi is the total number of papers co-authored by i with other institutions in the data set, and X is the total number of co-authorships between all the institutions in the data set.
Plainly speaking, it’s a measure of whether two institutions are doing more than collaborating than we might expect with each other relative to their tendency to collaborate with the other universities in the data set. If Pij > 1 , then universities i and j share more co-authorships than we expect relative to the other institutions in the data set, so we say they have a comparative preference for collaborating with each other. Conversely, Pij < 1 indicates that the two universities are doing less than we might expect.
Anyway. Here’s the NZ university network revised with the links now weighted by their corresponding RCP values:
Better. Here it’s apparent that AUT has a stronger link with Auckland Uni in addition to Lincoln and Waikato, and it should be pointed out that University of Auckland, AUT and Massey University are also closer to each other in the network, bearing in mind that all three have campuses within Auckland.
Now with a working measure, we move on to a larger sample. Bring on the Australians.
Clearly the Tasman Sea has a solid effect on the way New Zealand based researchers connect with those based in Australia; the links within the NZ cluster of universities have greater RCP weightings than those within the Australian cluster, implying a preference for domestic rather than trans-Tasman co-operation. Another feature to consider is that the Australian universities in the same states are grouped together, which is consistent with the idea that geographical proximity plays a significant part in a researcher’s choice of collaborator.
It would only be natural to wonder how academics interact on a global scale – do we ever grow out of talking almost exclusively to our friends and shun outsiders in some weird, grown up, Mean Girls-esque collection of cliques?
From observing how the Dutch and German institutions are grouped together, we might conclude that the language barrier is a large hurdle to overcome when jointly writing scientific literature – this also seems apparent from the Chinese-Hong Kong cluster, as well as Korean and Japanese institutions as well. But languages also tend to cluster geographically, so it is hard to disentangle the effect of language from distance.
It’s no question that with the constant progress of technology, connecting with people is becoming less costly. However, there are factors remaining that impede the prospect of a totally connected scientific community, some of which have been speculated on here. Of course pictures and hand waving don’t constitute a solid argument, but a thorough analysis of these factors and their effect on university collaboration will be in store for you, dear reader.
In the meantime, perhaps one should learn German, or Mandarin, or Dutch, or even Japanese. It’s not that hard.
About the data visualisations
In order to make the larger graphs efficient enough to be used in browser, the amount of connections a node could have to other nodes was limited to its top four RCP values. This change had no significant effect on the clustering observed when the full connection matrix was used. The change was only implemented for the QS, ANZAC and benchmark data sets.
Author
Bonnie Yu is a research assistant at Te Pūnaha Matatini and a member of Te Pūnaha Matatini’s Whānau group for emerging scientists. Her research projects focus on university collaboration networks.
The data visualisations of this post were prepared by fellow research assistant, Nickolas Morton.
Yarns of wisdom at Maths Craft Festival
Armed with knitting needles and crochet hooks, Drs Jeanette McLeod, Julia Collins and Phil Wilson are on a mission to bring maths to the masses.
The trio are behind Te Pūnaha Matatini’s Maths Craft Festival, running at the Auckland Museum on the weekend of September 3-4. The festival promises to get into the ‘knitty gritty’ of the maths behind craft and the craft behind maths with a range of hands-on activities and a series of public talks.
The mathematicians were inspired to start the festival after a serendipitous encounter while Julia was on holiday in Christchurch from Edinburgh. While visiting the University of Canterbury, Julia met Jeanette and discussed the mathematical things she had knitted to pass the time on her travels.
“Not only were these objects great ways to start conversations about maths with strangers who wanted to know what I was knitting, but making them also helped me to understand some deeper mathematics.”
Jeanette had a similar experience with crochet and knitting and the pair thought it would be great to offer the public a chance to learn not only about maths but different crafts as well. It wasn’t long before fellow mathematician Dr Phil Wilson became involved, bringing his experience with making origami models and his fascination with fractals.
Phil believes everybody will find something valuable at the Maths Craft Festival: “Teachers can get new ideas on how to engage their pupils with maths, knitters and crocheters will discover new and intriguing patterns, scientists and students can find new ways to explore shapes and patterns, and parents can relax with some zen colouring-in while their children try to see who can build the biggest fractal.”
Jeanette, Julia and Phil have planned a range of activities, including crocheting hyperbolic planes, building fractal sculptures, making Möbius strips, and folding origami dodecahedrons. Guest speakers will also be offering a series of maths and craft themed talks on topics from the mathematics behind knitting, to turning geometry into art.
“We hope that people of all ages and backgrounds will attend, no matter if they’re a novice knitter or a mediocre mathematician; a child in age or a child at heart,” Julia says.
The festival runs September 3-4 at the Auckland Museum. Visit www.mathscraftnz.org for further details.
What: Maths Craft Festival When: 3-4 September, 2016 – 10am – 5pm
Where: Auckland Museum
Cost: Free entry to festival with museum ticket
Web: mathscraftnz.org
#mathscraftnz
The festival is principally sponsored by Te Pūnaha Matatini, with further sponsorship provided by the University of Canterbury’s College of Engineering, the University of Auckland’s Mathematics Department and the Dodd-Walls Centre for Photonic and Quantum Technologies.
Reframing Innovation: conversations from around the web
Did you miss our Reframing Innovaton campaign with Figure.NZ? Catch-up on the conversation from around the web with our Storify:
Want to find more about telling stories with data? Sign up for a Figure.NZ account and start your own data board.
Cate Macinnis-Ng receives Plant Biology award
Congratulations to new Associate Investigator Dr Cate Macinnis-Ng who recently received the Roger Slack Award in Plant Biology. The annual award by the New Zealand Society of Plant Biologists recognises an outstanding contribution to the study of plant biology in the last five years.
Cate specialises in plant functional responses to environmental conditions. She is particularly interested in the impact of climatic variability and change on carbon and water fluxes of the forests. Since moving to New Zealand in 2010, Cate has been exploring the physiology of our largest and longest-lived tree, kauri. Among key findings Cate has found evidence for drought adaptations in a species previously thought to be highly vulnerable to water stress. This research has laid the groundwork for a Rutherford Discovery Fellowship exploring limits of drought tolerance using a NZ-first throughfall exclusion experiment. Cate will be presenting her work at the upcoming NZSPB annual meeting at Queenstown Research Week on September 2nd.
Māori Knowledge & Development Panel Forum
He aha te kai a te rangatira? He Kōrero, he kōrero, he kōrero. Nō reira, nau mai, haere mai!
You are warmly invited to attend a PBRF forum focused on the Māori Knowledge & Development Panel led by Professor Margaret Mutu and Dr Aroha Harris.
The forum is:
- Aimed at clarifying MK&D panel criteria
- Offering guidance, tips and suggestions from our Panel experts
- Chaired by Dr Melinda Webber (AD PBRF Faculty of Education and Social Work)
Date: Monday, 22 August 2016
Time: 4.00 – 5.30 pm (Drinks and nibbles from 5.00pm)
Venue: Women’s Federation Room, Old Government House
Please register your interest by emailing Emma on e.buchanan@auckland.ac.nz by 18 August for catering purposes.
A joint initiative of the University of Auckland’s Faculty of Arts and Faculty of Education & Social Work.
The (my) future and other predictions with greater than 5% error
What are you going to do after you finish your PhD? Where do you want to go? Are you going to become a lecturer? These are all questions that I field on a regular basis. Rather than going with my instinctive response of “What the hell? I don’t even know what my PhD is about yet!”, I usually say something like “I don’t know, but hopefully something in conservation or consulting”. Apparently this puts me in the minority of PhD students in that I do not desire to go into academia.
This was a topic discussed at the New Zealand Association of Scientists conference I attended on the 26th April; you can also read about it in my previous blog post. One of the speakers referenced the Royal Society report where it stated that while about half of PhD students continue on with research, becoming early career researchers, most end up leaving academia for work in industry. This is despite most PhD candidates desiring a job in academia at the outset of the project. The question asked at the conference is how can we, as the scientific community, support PhDs and Post Docs so that if an academic career does not pan out they can successfully and relatively painlessly transition into industry? As one of the members of the emerging researchers panel said, when she was faced with the current situation, it is not unusual to feel like the best option is just to “give up”.
I am lucky in that I have a great team of supervisors (I have 4 ± 1 supervisors) who want my PhD to be more about preparing me for future work rather than me just churning out papers. They have suggested that I take opportunities to learn skills that will be useful in industry and that I take time to build connections inside and outside of academia. However, not everyone is as lucky in having such excellent supervisors. I have heard horror stories about supervisors who refuse to meet with their students and those who take no role in preparing the student for the future. What can we do for these students without supervisor support?
This is a place where student-led organisations can step in. The Te Pūnaha Matatini Whānau committee is well aware of these trends and are currently working on a number of projects to address this. The Whānau has connected with industry partners such as data analytic companies. The intention is for TPM Whānau members to be eligible to undertake internships at the companies. This will teach the members new skills and give experience that will be valuable in industry. We are also organising a data debate on the issues of data privacy between industry members and Te Pūnaha Matatini.
Ultimately however, no matter how supportive the supervisor is, it is up to the student to make sure that they obtain the experience and skills they need. As one of my supervisors said, “if you are smart enough to get to PhD level you are smart enough to look after yourself”.
With that I will sign off and go look after myself.
Jonathan
About
Jonathan Goodman is a Te Pūnaha Matatini Whānau committee member.