When There Is Not Enough Evidence and When Evidence Is Not Enough: An Australian Indigenous Smoking Policy Study

Vujcich, Daniel; Rayner, Mike; Allender, Steven; Fitzpatrick, Ray

doi:10.3389/fpubh.2016.00228

ORIGINAL RESEARCH article

Front. Public Health, 20 October 2016

Sec. Public Health Policy

Volume 4 - 2016 | https://doi.org/10.3389/fpubh.2016.00228

This article is part of the Research Topic Evidence-based public health: why, what and how View all 5 articles

When There Is Not Enough Evidence and When Evidence Is Not Enough: An Australian Indigenous Smoking Policy Study

$\r\nDaniel Vujcich*$ Daniel Vujcich¹*

Mike Rayner²

Steven Allender³ $Ray Fitzpatrick\r\n$ Ray Fitzpatrick²

¹Western Australian Department of Health, Public Health Division, Perth, WA, Australia
²Nuffield Department of Population Health, University of Oxford, Oxford, UK
³School of Health and Social Development, Deakin University, Geelong, VIC, Australia

Background: The Indigenous Tobacco Control Initiative and Tackling Indigenous Smoking Measure were both announced by the Australian Government at a time when its rhetoric around the importance of evidence-based policy making was strong. This article will (1) examine how the Rudd Government used evidence in Indigenous tobacco control policy making and (2) explore the facilitators of and barriers to the use of evidence.

Methods: Data were collected through (1) a review of primary documents largely obtained under the Freedom of Information Act 1982 (Commonwealth of Australia) and (2) interviews with senior politicians, senior bureaucrats, government advisors, Indigenous health advocates, and academics. Through the Freedom of Information Act process, 24 previously undisclosed government documents relevant to the making of Indigenous tobacco control policies were identified. Interviewees (n = 31, response rate 62%) were identified through both purposive and snowball recruitment strategies. The Framework Analysis method was used to analyze documentary and interview data.

Results: Government policy design was heavily influenced by the recommendations presented in government authored/commissioned literature reviews. Resulting policies were led by equivocal evidence for improved tobacco control outcomes among Indigenous Australians. Many of the cited studies had methodological limitations. In the absence of high-quality evidence, some policy makers supported policy recommendations that were perceived to be popular among the Indigenous community. Other policy makers recognized that there were barriers to accumulating rigorous, generalizable evidence; in the absence of such evidence, the policy makers considered that the “need for action” could be combined with the “need for research” by introducing innovative strategies and evaluating them.

Discussion: Despite the absence of high-quality evidence, the formulation and adoption of Indigenous tobacco policy was neither irrational nor reckless. The decision to adopt an innovate and evaluate strategy was justifiable given (a) the potential for the gap between Indigenous and non-Indigenous health outcomes to worsen in the absence of an imminent policy response; (b) the existence of circumstances, which made it difficult to obtain high-quality evidence to guide policy; and (c) the need for policy solutions to reflect community preferences, given sociohistorical sensitivities.

Introduction

“Evidence-based policy” has become a synonym for “good policy,” prompting the observation that “[i]t is difficult to imagine anyone arguing that policy should be based on anything but the best available evidence” (1). In a critical analysis of literature relating to the use of evidence in health policy making, Oliver et al. noted that the majority of studies made normative assumptions that policies should be based on evidence, and that any barriers between evidence and policy ought to be overcome (2). However, few of the studies used empirical data to demonstrate why evidence-based policy making is superior to other forms of decision-making. Similarly, only a small number of studies provided in-depth empirical descriptions of the importance of evidence relative to other considerations.

Numerous models and theories conceptualize evidence as just one part of a broader scheme of competing influences on policy. For instance, Walt and Gilson considered that the content of health policies is affected by an interplay between context (e.g., culture, economic factors, demography, history, ideology), actors (i.e., individuals, groups, or organizations and their position in the power hierarchy), and the process through which policy issues are communicated, negotiated, and decided (which can include the consideration of evidence) (3, 4).

A contemporary adaptation of this model is Lin’s description of health policy as the product of three competing rationalities (5). “Cultural rationality” is defined as “values, ethics, what (perceived) societal opinions feel is right in relation to health policy” and thus aligns with what Walt and Gilson term “context” (5). “Political rationality” relates to the process through which power is exercised and decisions are made and includes such factors as “the willingness of policymakers to have transparent processes and be accountable, the ability of interest groups to participate … and the role of commentators (be it media, experts, or lobbyists)” (5). Finally, “technical rationality” describes the knowledge produced by researchers and can include diverse forms of evidence, such as epidemiology and economics (5). Lin argues that these rationalities (and the policies that they create) are shaped by “historical political legacies” and reflect “ongoing processes of social learning” (5).

In view of such rich theoretical explanations, Oliver et al. call for a new research agenda, which focusses on the “influences on and processes of policy” through in-depth, empirical descriptions of how evidence “fits with the other drivers and triggers that affect policy” (2). This article uses the development of tobacco control policies for Indigenous Australians as a case study for examining the real-world tensions inherent in the health policy-making process.

When the Labor Party assumed government in 2007 under the leadership of Prime Minister Kevin Rudd, Indigenous Australians were expected to have lives that were 17 years shorter than their non-Indigenous counterparts (6). The Labor Party argued that such enduring disparities were a product of ineffective, ideologically driven policies and that “from here on our guiding principle will be the evidence of what works and what does not work in reducing disadvantage” (7).

The Indigenous Tobacco Control Initiative and Tackling Indigenous Smoking Measure committed over AUD 120 million in government funds and marked the beginning of a period “that has seen more action on Indigenous smoking than any other time in our history” (8). Yet, perhaps surprisingly, in light of the foregoing context, it was said of some aspects of the policies that “the government hasn’t produced evidence to back up its campaign” (9).

This article will (1) examine how the Rudd Government used evidence in Indigenous tobacco control policy making and (2) explore the facilitators of and barriers to the use of evidence relative to other factors.

In so doing, this article reveals difficulties in the application of evidence-based policy making. These difficulties include debates as to what constitutes “evidence” and a tension between the need for good evidence and the need for urgent policy action. The article also demonstrates how, in some circumstances, the need for evidence of effectiveness can be exceeded by the need to empower the target population in the decision-making process.

Materials and Methods

Data were collected through a combination of document reviews and interviews. The overarching purpose of data collection with respect to each method is set out in Table 1 below. Data collection and subsequent analysis were informed by awareness of the possibility that key factors might turn out to relate to “context/cultural rationality” and “political rationality/process” as much as “technical rationality.”

TABLE 1

Table 1. Summary of goals of data collection, by method.

It is considered prudent to combine these two methods when conducting policy process analyses to mitigate their respective limitations (4, 10). These limitations include the fact that (1) some important events in policy processes are often not recorded, or the documents may be withheld from public access; (2) documents can possess official and perhaps incomplete versions of events; (3) respondents may underrepresent or overrepresent their role in a policy process; and (4) the passage of time might impede a respondent’s ability to recall events accurately or with sufficient detail (10).

Documentary Data Collection

A two-stage process was adopted to identify relevant documents. In the first instance, publicly available documents were obtained by searching the following websites/databases:

• Department of Health and Ageing (DoHA) (“Publications” and “Media Release and speech archive” sections only);

• Australian Indigenous HealthInfoNet (“Policies” and “Tobacco” sections only);

• Tobacco Control Supersite (“Australian Tobacco Timeline” section only);

• National Aboriginal and Torres Strait Islander Health Equality Council;

• National Preventative Health Taskforce;

• Australian National Council on Drugs;

• Ministerial Council on Drug Strategy;

• Intergovernmental Committee on Drugs;

• Quit;

• Factiva (Australian national newspapers).

Where search functions were available, the search was limited to items containing the search terms “(Aboriginal OR Indigenous) AND (tobacco OR smok*).” Where search functions were not available, items were individually screened by title for relevance. Relevant documents were limited to those produced between 2007 (election of Rudd Government) and 2008 (announcement of policies).

The second-stage of the document review was focused on obtaining relevant documents produced for the private use of members of the policy-making community. These documents were either provided by the interview respondents or accessed under the federal Freedom of Information Act 1982 (FOI).

The initial request (FOI Request 309-112) for “written reports/letters of advice and internal reports or memoranda used by the Government to inform the design of the Indigenous Tobacco Control Initiative” resulted in an estimated charge of AUD 10,845 for the time that FOI officers and document custodians would likely spend searching for, retrieving, and assessing the documents. Further particularized requests were then submitted (FOI Requests 006-1314 and 067-1213) seeking access to the following:

• minutes of meetings held between December 2007 and 20 March, 2008 in which the Indigenous Tobacco Control Initiative was discussed;

• minutes of any meetings relevant to the selection of the first six multicomponent projects funded as part of the Indigenous Tobacco Control Initiative in 2008–09;

• minutes of meetings held between December 2007 and December 2008 in which the tobacco elements of the National Partnership Agreement on Closing the Gap in Indigenous Health Outcomes were discussed;

• any research commissioned before December 2008 to inform the Tackling Smoking Measure specifically;

• memoranda or advice prepared between December 2007 and 20 March, 2008 in which the Indigenous Tobacco Control Initiative is mentioned;

• memoranda, advice, or briefing papers drafted before December 2008 in which the Tacking Indigenous Smoking Measure is mentioned.

The revised requests yielded 24 previously undisclosed documents relevant to the making of the policies. One document was not disclosed on the basis that it was produced for Cabinet deliberations and therefore exempt under the legislation.

Interview Data Collection

Interview participants were recruited through a combined purposive and snowballing sampling strategy, as recommended for studies of this nature (10, 11). A preliminary list of individuals involved in the policy-making process was created using the documents described above. These purposefully selected respondents were asked to name other people who influenced the policy process in relation to Indigenous tobacco control.

Fifty potential respondents were invited to participate in interviews. A source population of 50 policy actors was considered reasonable, given that the study was limited to two specific policies. Thirty-one individuals agreed to participate in the study, representing a response rate of 62%. The response rate is consistent with those obtained in other studies of the public services sector (12).

The sample comprised advisors to the Federal Minister for Health (n = 2), senior Federal politicians (n = 2), senior Federal health bureaucrats (n = 4), members of and assistants to the National Preventive Health Taskforce and its Tobacco Working Group (n = 9), members of the National Indigenous Drug and Alcohol Committee (n = 2), researchers/academics (n = 4), and Indigenous health advocates (n = 8). Anonymity was guaranteed to protect reputations and interests. Of those individuals who did not respond to or declined interview invitations, four were politicians, seven were government advisory group members, one was a senior bureaucrat, four were researchers, and three were Indigenous advocates.

Interviews were audio recorded and transcribed. Interviews were semi-structured, using a combination of open-ended and probing questions. Topic guides were prepared before each interview and were tailored to the knowledge and experience of each respondent. Where relevant, respondents were asked to describe the following:

• factors that might explain the policy attention given to Indigenous tobacco control under the Rudd Government;

• how the policies were developed;

• factors that might explain why certain approaches to tobacco control were prioritized over others;

• the role of evidence in the policy process;

• facilitators of and barriers to the use of evidence.

Ethics approval was granted by relevant committees at the University of Oxford (Ref: SSD/CUREC1/12-013) and Deakin University (Ref: 2012-218).

Data Analysis

Data were analyzed using the Framework Analysis method. The process involved (1) becoming familiar with the data by listening to/transcribing the recordings and reading the transcripts and documents; (2) noting down recurring themes and subthemes (see Table 2 below); (3) using QSR International’s NVivo 10 software to code the data according to the themes; (4) transferring the coded data into charts (one for each theme), which were organized in rows by source and in columns by subtheme; and (5) looking down the chart columns to observe recurring patterns and tensions.

TABLE 2

Table 2. Themes and subthemes coded for data analysis.

Results

The Role of Evidence in Getting Indigenous Tobacco Control on the Agenda

There had long been evidence to indicate high rates of Indigenous smoking. A 1994 survey of over 15,700 Indigenous Australians found that 49.7% of respondents over the age of 13 years smoked daily (13). Similarly, a survey conducted a decade later (2004–05) found that 50% of Indigenous Australians aged 18 years and over (n = 10,439) were current daily smokers, compared to 23% of the general population (14).

Despite these data, there had been an absence of significant government investment specifically geared toward reducing Indigenous smoking rates prior to 2008 (15). For instance, an expenditure analysis revealed that smoking was a focus in only 2 and 3% of Indigenous-specific alcohol and other drug intervention projects in 1999–2000 and 2006–2007, respectively (16).

The interview findings suggest that the increased political attention given to Indigenous smoking in 2008 was the product of a confluence of three factors – namely – (1) the publication of evidence showing that tobacco was the single largest contributor to the “gap” between Indigenous and non-Indigenous health outcomes; (2) the success of the Closing the Gap advocacy campaign, which was led by Indigenous and civil society groups and supported by the incoming Labor Government; and (3) the appointment of a Health Minister committed to tobacco control, perhaps best evidenced in the Rudd Government’s decision to make Australia the first country in the world to introduce a law to mandate plain cigarette packaging.

Burden of Disease Study

Six respondents cited a burden of disease study by Vos et al. published in 2007 (hereafter, the Vos study) as an explanation for why Indigenous smoking became more politically prominent in 2008 (17). The study found that, of 11 risk factors, tobacco was the largest single contributor to both the overall Indigenous disease burden (12%) and the health “gap” between Indigenous and non-Indigenous Australians (17%) measured in disability-adjusted life years.

One senior bureaucrat said of the Vos study, “the study was so important.” The significance of the Vos study was that it “show[ed] people what were the contributing factors to excess mortality” (senior bureaucrat). An Indigenous leader and health advocate, recalled that Vos’ work prompted an Indigenous research organization to focus on smoking:

[The Cooperative Research Centre for Aboriginal and Torres Strait Islander Health] put briefings together and all that and started to tell their story to the Minister … And then they did this presentation in Parliament House. Within a fortnight [the Government] found $14 million.

The presentation to which the respondent referred formed part of the Parliamentary Showcase of Aboriginal Health Research, which was held in March 2008 and opened by the Health Minister. As part of the program, Viki Briggs of the Centre for Excellence in Indigenous Tobacco Control (CEITC) delivered a presentation entitled “Reduce Smoking, Reduce the Gap” in which the findings of the Vos study were highlighted (18). Another respondent considered that “that was probably an absolutely crucial event” (Indigenous tobacco control academic). A Ministerial advisor considered that the event “helped crystallise the view that tobacco was a critical factor.”

The “Closing the Gap” Movement

Four respondents were of the view that the impact of the Vos study must be considered in its sociopolitical context. The study was published in 2007 at a time when the Closing the Gap campaign for Indigenous and non-Indigenous equality was gaining momentum (19). In the lead up to the 2007 Federal election, the Labor Party made a commitment to improve Indigenous social and health outcomes in response to the Closing the Gap campaign.

Evidence demonstrating the impact of smoking on the gap between Indigenous and non-Indigenous health outcomes found fertile ground in this sociopolitical landscape. According to a senior politician:

[W]hile the strong research helped galvanise people around [Indigenous tobacco control] I don’t think it was the cause or catalyst for action … [T]he very successful campaign around Close the Gap … helped create political momentum and interest – the research is vital in this context but could’ve gone unnoticed without the broader campaign.

Similarly, a senior bureaucrat explained that a “catalyst” for political action “was that we had a figure, a statistic that said smoking causes 20% of all mortality. It came at the same time as Closing the Gap. So, okay, to close the gap … you’ve got to prevent premature mortality.” A Ministerial advisor agreed that “we were never going to achieve this [aim to close the gap in health outcomes] if we didn’t do something about Indigenous smoking.”

Political Leadership

While the confluence of evidence and political agendas created a positive environment for Indigenous tobacco control to become more politically salient, four respondents emphasized the crucial role of the Health Minister in actually securing policy action. It was noted that DoHA’s interest in Indigenous tobacco control predated the policy developments in 2008 but that, despite this, the issue never gained political attention. According to a senior bureaucrat, “[w]henever we put a case around a smoking intervention, every minister would say, ‘yeah, well who the hell’s going to run it because they [Aboriginal health staff] all smoke anyway?’”

By contrast, the new Health Minister, Nicola Roxon, was described as someone who understood and championed the importance of Indigenous tobacco control. A Government advisor explained that the “bottom line is Nicola Roxon … got it and put money into it” (Tobacco Working Group member). Similarly, an Indigenous leader and health advocate was of the view that “you can put it [policy action] down to a couple of people in the Minister’s office and the Minister herself. She was committed to saying, ‘if that’s what the evidence is telling us, we need to put a bigger effort into here.”

Role of Evidence in the Design of the Indigenous Tobacco Control Initiative

Both interview data and documents obtained under FOI confirmed that DoHA played a lead role in the design of the Indigenous Tobacco Control Initiative, which occurred over a short period of time due to political and organizational pressures. A Ministerial advisor noted “there was a sense that after the Apology [to the Stolen Generations in February 2008] that … [the] goodwill created by that needed to be translated into some sort of firm commitments” (Ministerial advisor).

The Indigenous Health Equality Summit was scheduled to take place in Parliament House in March 2008 and was expected to culminate in the signing of a partnership agreement between government and non-government organizations to achieve health equality between Indigenous and non-Indigenous Australians. The interview data suggested that the Prime Minister had expressed an intention to attend the event, and that this served as a catalyst for the development of the policy. A Ministerial advisor said:

I do remember this being an example where we were kind of going, ‘Ergh! I think the PM might want to announce something’ … [A]s I say, it was all a little bit chaotic in those early days. So it meant that a lot of the sort of detailed policy development was done at the Departmental level more so than at the Ministerial level.

Media reports, to which the respondent referred, revealed that the Health Minister had been dissatisfied with unrealistic time pressures imposed by Prime Minister Rudd (20). According to the Ministerial advisor, “the Department [of Health and Ageing] sort of came up with this bunch of proposals and said, ‘look … we’ll give you something to announce at this event.’”

The Indigenous Tobacco Control Initiative was announced by Prime Minister Rudd at the Indigenous Health Equality Summit. Through the Initiative, the Government committed AU$14.5 million over 4 years to:

• train Indigenous health staff in smoking-cessation strategies;

• trial innovative community interventions, including culturally appropriate communication activities, in five or six pilot sites; and

• support Indigenous tobacco control research to “help build the evidence base around what works” (21).

One of the documents obtained under FOI was a literature review (dated June 2007) authored by an employee of DoHA Program Management and Evaluation Unit. The review proposed five policy options, four of which seem to have been incorporated into the Initiative developed by DoHA – namely:

• offer support and training for local health staff to deliver brief tobacco cessation interventions and programs that include nicotine replacement therapy;

• provide more intensive interventions in targeted communities/regions;

• develop media campaigns featuring local identities or artwork;

• fund the CEITC to audit evaluations of tobacco control initiatives.¹

Table 3 below contains the studies cited in DoHA review to support each policy proposal and provides a summary of their major findings and relevant limitations.

TABLE 3

Table 3. Summary of studies cited in DoHA (see text footnote 1) review in support of policy proposals.

Role of Evidence in the Design of the Tackling Indigenous Smoking Measure

The second major Indigenous tobacco control policy – the Tackling Indigenous Smoking Measure – formed part of the Council of Australian Governments’ National Partnership Agreement on Closing the Gap in Indigenous Health Outcomes, which was signed in December 2008 (29).

The interview data suggest that DoHA played a key role in the development of this policy also. According to a Ministerial advisor, “the Department developed the proposals that we kind of further refined in cooperation with them I guess.” Interviews with DoHA staff confirmed the central role of the Department in designing the policy and also highlighted the informal influence of external advisors. A senior bureaucrat recalled:

In the early stages of [determining] ‘well, what do we do?’ we talked to the Centre for Excellence in Indigenous Tobacco Control [CEITC] … and we had conversations with them initially about ‘well, we’ve got money to do X and Y and Z, if you could do something what do you think needs to be done?’

DoHA also commissioned CEITC to draft a literature review and a scoping paper to inform its response to Indigenous smoking rates. The resulting paper was provided to DoHA in September 2008.

Additionally, advice on the formulation of the policy was received from the National Preventative Health Taskforce. The National Preventative Health Taskforce was established by the Government to assist in the development of strategies to prevent diseases related to tobacco, alcohol, and obesity. While the Taskforce did not release its final recommendations until 2009, a Government report revealed that interim advice was provided in 2008 (30). According to a member of the Taskforce’s Tobacco Working Group, “[t]hey involved us in the development of the Tackling Smoking Initiative … We were doing the same recommendations formally and informally.” The Director of CEITC was a member of the Tobacco Working Group.

The formal recommendations published by the Tobacco Working Group reproduced almost verbatim the recommendations contained in the literature review and scoping paper drafted by CEITC for DoHA (31). The relevant recommendations in the CEITC paper were to

• increase the Indigenous tobacco control workforce;

• improve health worker access to appropriately designed training;

• improve access to pharmacotherapies;

• develop a well-researched and focus-tested Indigenous-specific social marketing campaign at national and local levels;

• raise the profile of Quitlines to Indigenous communities (32).

These recommendations closely aligned with the Tackling Indigenous Smoking Measure, which committed AUD 100.61 million to reduce Indigenous smoking rates and the burden of tobacco-related disease through a number of initiatives, the most prominent of which were the following:

• establish a national network of tobacco action coordinators;

• develop a national Indigenous tobacco action training program for health workers and community educators;

• strategies to improve delivery of smoking-cessation services, including nicotine replacement therapy;

• social marketing campaigns to reduce smoking-related harms among Aboriginal and Torres Strait Islander peoples;

• enhance Quitline to provide culturally sensitive services (29, 33).

Table 4 below summarizes the evidence cited in the CEITC paper to support each policy proposal and notes major findings and limitations of each.

TABLE 4

Table 4. Summary of studies cited in CEITC paper in support of policy proposals.

Rationale for Adopting Policy Proposals in the Absence of Strong Evidence

It is clear from Tables 3 and 4 that not all of the policy initiatives were grounded in strong evidence demonstrating their likely effectiveness. A critical analysis of the studies revealed that many had limitations, such as inadequate reporting of methods, selection bias, response bias, and small sample sizes. Most Indigenous Australian studies measured subjective preferences for interventions, perceptions of likely impact of interventions, or thoughts and attitudes about smoking. In the absence of rigorous evidence from the Australian context, some reliance was placed on studies of indigenous populations in other countries. Interviews were used to understand why the policy makers adopted the policy proposals in the absence of clear supporting evidence.

Impractical to Wait for Rigorous Indigenous-Specific Evidence

Faced with both a dearth of evidence and an understanding that Indigenous smoking rates were inexorably high despite the presence of mainstream tobacco control strategies, some policy makers saw the need to take action based on inference. An Indigenous tobacco control academic with close links to key policy makers noted that “the problem is we didn’t have any evidence from this [Indigenous] setting so … people had to make guesses. And they probably weren’t bad guesses.” A Tobacco Working Group member used similar language, describing the research conducted to develop policy priorities as “predominately guess-based.”

An innovate and evaluate approach was accepted as being an appropriate solution in the circumstances. Internal documents obtained under FoI demonstrate the extent to which policy makers considered that it was both necessary and justifiable to act without a strong evidence base, so long as there was a commitment to accumulating knowledge in the process:

Currently there is little available evidence for what works in Indigenous tobacco control. In partnership with research bodies … the Government can make an immediate start on building that evidence-base …

In tandem with the research program, it is proposed to run five or six pilot projects in selected locations to trial innovative approaches to smoking prevention and cessation … Evaluation of these pilots could inform future phases of the initiative …

The approach recommended here combines action (the pilot programs) coupled with a strong research focus to ensure that resources are not wasted on misdirected initiatives.²

The alternative approach – that is, waiting for evidence to emerge before taking policy action – was not considered practical. Two policy makers noted that the Indigenous context presented distinct challenges to the conventional understanding of evidence-based policy making, arguing that evidence needed to play more of a back-end role. According to a Tobacco Working Group member:

[W]e need to look at ways of finding types of intervention that would resonate better with Aboriginal communities and I think a part of what came with that [was] also an acceptance that certainly when it comes to mainstream programs, it is driven by research … but … when it comes to Aboriginal smoking there’s a need to be willing to take a few more risks, and trial different ways of addressing the problem.

Researchers with links to policy makers expressed the view that the Indigenous research context presented distinct challenges. One respondent offered the following example of the limitations:

[T]he logistics of doing high level randomised controlled trials in [remote] populations … it’s ridiculous … There were a whole lot of issues with cross-contamination between communities … We did the numbers for a randomised control trial in giving quit advice … the numbers to do a proper randomised control trial in the Northern Territory were going to be 30,000 participants. You couldn’t get it … (Indigenous tobacco control academic).

An Indigenous advocate with research experience added:

[T]here’s always challenges from the remoteness of going out to do the pre- and post-questionnaires … a lot of these people move around in their communities … They will travel around and visit family and the neighbouring community. Could be anything … even the weather … From all those kind of logistic things, to even language being a barrier.

Policy Proposals Were Championed by Indigenous Representatives

Not all members of the policy community subscribed to the view that mainstream tobacco interventions were ineffective for Indigenous people, or that the Indigenous-specific interventions recommended in the CEITC paper were needed. One Tobacco Working Group member warned against “overdoing the differences” and noted that “Aboriginal people are people. They smoke like other people. They quit like other people. And I think we need to be almost wary of assuming that you need to have something incredibly and dramatically different.” Another Working Group member concurred, arguing that “there is absolutely zero evidence that those [mainstream] approaches do not impact Indigenous people in the same sort of was as they affect the rest of us.” Longitudinal studies lend some support for this view showing that, in the absence of major Indigenous-specific policies, daily smoking prevalence for Indigenous males over the age of 18 years decreased from 58.5% in 1994 to 52.6% in 2008 with a statistically significant annual decrease of 0.4% per year (52).

The apparent mismatch between the advice provided to Government and the view of some respondents that some Indigenous-specific interventions were not necessary was explained in the following way:

I know there would have been at least two or three people around the table who would have shared what I just said to you about ‘oh, god, do we really have to go down this … culturally distinctive route?’ But I think in the scheme of things we didn’t feel that strongly about it to say, ‘listen, let’s just can that.’ Because I think that critique needs to come out of the Indigenous community rather than be given like Moses from above” (Tobacco Working Group member).

An expert government advisor agreed that, in light of the history of exclusion, Indigenous perspectives need to be incorporated into the policy process:

… [P]eoples outside of mainstream Australia have a right and should be involved in leading the progression of policy development around those marginalised populations … [I]t’s time for the policy developers in Australia to now say to Aboriginal Australians, ‘you must develop the policies and you must assist us on the pathways out’ (National Indigenous Drug and Alcohol Committee member).

Other members of the Tobacco Working Group confirmed the importance of ensuring that the policies were developed with and supported by Indigenous people:

[B]ecause of the awful history for Aboriginal people in this country there’s a real strong desire amongst representatives of the Aboriginal community that you’ll be working with … wanting it to be Aboriginal-determined, controlled etc.

[A] lot of Aboriginal people were really keen on the idea of local workers. Would it be in my core program? Not necessarily. But if it’s the price we pay for getting Aboriginal people [to] support the rest of the program – willingly pay it.

Discussion

This case study has demonstrated empirically what the theoretical models of policy making suggest – namely – that evidence does not exist and cannot be used in a vacuum. Rather than viewing evidence as being in competition with other policy factors, this case study presents evidence as being dependent on them.

The intuitive appeal of evidence-based policy making lies in the perception of objectivity – evidence “speaks for itself” (53). Certainly, the data suggested that evidence played an important role in getting Indigenous smoking onto the political agenda. The salience of the Vos burden of disease study was repeatedly described in interview responses and relevant documents (17). However, the findings show that while “[s]cience may be able to deliver empirical knowledge … this knowledge has to be interpreted and warranted in order to be understood” (54).

In this instance, there were two factors that enabled the evidence to “speak” in the policy process. First, the evidence emerged against the backdrop of a popular sociopolitical campaign to reduce health disparities between Indigenous and non-Indigenous Australians; this aligns with both Walt and Gilson’s notion of “context” and Lin’s definition of “cultural rationality” as “what (perceived) societal opinions feel is right” (3, 5). Second, the evidence was understood and valued by key figures in Government (including the Minister for Health) who had the power to take action in response; this reflects the importance of “actors” and “process” in Walt and Gilson’s model, and “political rationality” in Lin’s schema (3, 5).

The symbiotic relationship between evidence and other policy factors was further demonstrated in the policy design stage. The recommendations presented in the government authored/commissioned literature reviews (obtained under FoI) seem to have heavily influenced the content of the policies. In relation to the Indigenous Tobacco Control Initiative, “political rationality” and “context” seem to be major explanations for the fact that most of the recommendations emerging from the literature review were adopted in the policy. The political pressure to make a major policy announcement at the Indigenous Health Equity Summit meant that policy proposals needed to be developed by DoHA within a short period of time. In this situation, the importance of the evidence seems, at least in part, to have been associated with its accessibility (the literature review was drafted internally by the Department in the previous year). The accessibility of research has been identified as a facilitator to evidence-based policy making in other studies (55–58).

Similarly, the Tackling Indigenous Smoking Measure closely reflected the recommendations in another literature review commissioned by DoHA and authored by the CEITC. Viewed through the prism of Walt and Gilson’s model of health policy making, the value that was placed on the CEITC’s research can be partly ascribed to the organization being regarded as a key “actor” in the policy community. If the policy community is conceptualized as a network of interconnected individuals and organizations, CEITC’s influence derived from its centrality in that network (59). DoHA provided CEITC with a majority of its funding with a view to building the evidence base in relation to Indigenous tobacco control and, as demonstrated in the interview data, relied on the organization for policy advice. Furthermore, the Director of CEITC was a member of the Tobacco Working Group, which was also consulted in the development of the policy. From the perspective of Lin’s model, “political rationality” overlapped with “technical rationality” in the sense that a research organization was invited into the decision-making process.

While it is clear that the tobacco control policies were developed based on evidence in the form of literature reviews, a critical analysis of the studies contained in those literature reviews revealed that many possessed methodological limitations. The evidence relied on was equivocal as to whether the recommendations would lead to positive tobacco control outcomes for Indigenous Australians. The case study therefore highlights the importance of conceptual clarity when using the term “evidence-based policy.”

However, it does not follow that policies based on weak or equivocal evidence are “bad policies.” Far from being irrational or reckless, the interview data suggest that the policy makers’ acceptance of the policy proposals was based on logical reasoning, notwithstanding the absence of a rigorous evidence base.

Respondents noted the importance of other influences on the policy process, including the need to ensure that the policies had the support of members of the Indigenous population. Historically, policy making for Indigenous Australians has often been a top-down exercise associated with marginalization and disempowerment of the population. For instance, an inquiry into a controversial 2007 Indigenous policy initiative (the Northern Territory National Emergency Response) found that the policy “diminished its own effectiveness through its failure to engage constructively with the Aboriginal people it was intended to help” (60). For some of the policy makers in the present case study, there was a desire to avoid repeating mistakes of the past by ensuring that the needs and preferences of Indigenous people were captured in the design of the Indigenous Tobacco Control Initiative and Tackling Indigenous Smoking Measure.

Evidence cannot trump all other factors; rather it must coexist with other inputs in the policy process, including the “thoughtful identification and compassionate use of individual patients’ [or populations’] predicaments, rights and preferences” (61). This can be illustrated by another example from the Australian Indigenous health context. While there is a strong evidence base to support the use of dialysis for patients with renal failure, historically, many Indigenous patients living in remote areas refused treatment on the basis that it would involve spending extended periods of time in urban and regional centers, away from their family, culture, and land (62). Put simply, “an efficacious treatment cannot produce public health benefits if it is not adopted widely by its intended recipients” (63).

Ideally, solutions that are “acceptable” to the target population will also be supported by evidence of their likely effectiveness. However, the Indigenous tobacco control case study demonstrates that there are situations in which it may be justifiable to introduce policies in the absence of high-quality evidence.

It has been noted that barriers to research in Indigenous contexts include

• the fact that the Indigenous Australian source population is small (670,000), meaning that sample populations in Indigenous-specific studies are also likely to be small, and only very large program impacts are likely to be statistically significant (64);

• Indigenous distrust of research on the basis that it was historically associated with “the politics of colonial control” and, in the present day, can be perceived to be exploitative, stigmatizing, or unresponsive to Indigenous needs and concerns (65–67); and

• the fact that, in order to be culturally sensitive, there is often a call for research in Indigenous communities to be participatory, which can result in a conflict between “the values of the academic setting and those of the community” (65, 68).

The consequence is that “‘best’ evidence is often gathered on simple interventions and from groups that are easy to reach in a population,” and evidence from those that are harder to reach tends to be regarded as inferior (69).

This research bias is not unique to the Indigenous Australian context. It has been noted that low-income countries often have limited resources to invest in research, and cultural minorities within high-income countries are often excluded from research protocols because of the logistical and financial costs, such as those related to translation (70). With respect to tobacco control research, a systematic review of studies examining the effectiveness of NHS stop smoking services in the United Kingdom found there is currently a lack of high-quality research into the potential differential impact of interventions in other subpopulations, such as black and minority ethnic groups, despite qualitative and anecdotal evidence suggesting different beliefs and attitudes to smoking and smoking cessation (71).

Birch presents the potential resulting paradox of a “no action without supporting evidence” mentality thus:

If there is no evidence of effectiveness among the most deprived group… [e]fficient use of scarce resources… would imply the intervention not be given to these individuals with the anticipated increase in social inequalities in health. The resources of a health care system would become increasingly concentrated on less deprived populations. This results not necessarily because the problems of the more deprived groups are insoluble, but because the mechanisms we choose to evaluate for dealing with the observed problems are particularly suited to less deprived groups, i.e. it is a feature of the ‘service focus’ (72).

Applied to the Indigenous tobacco control context, a rigid adherence to the belief that there should be “no policy without evidence” could feasibly have resulted in a widening of the health gap between Indigenous and non-Indigenous Australians simply because there was more research to demonstrate the effectiveness of interventions in the latter population.

When the potential corollaries of the “no policy without evidence” approach are laid bare, the policy makers’ decision to adopt the Indigenous Tobacco Control Initiative and Tackling Indigenous Smoking Measure in the absence of strong evidence is justifiable. The policies were not made with reckless disregard for what would work; instead, evidence from other settings and expert opinions were used to generate conceptually plausible responses to the policy problem in a manner that accommodated the apparent preferences of members of the Indigenous community. Moreover, the policy proposals were accompanied by a commitment to evaluate their effectiveness. This approach is consistent with that advocated by Sanderson, who suggests that policies should be viewed as hypotheses to be tested through pilots or trials and/or rigorous monitoring and evaluation (73).

In May 2015, the Government committed a further AUD116.8 million to Indigenous tobacco control initiatives over 3 years (74). Future research needs to examine whether rigorous evaluations of the Indigenous Tobacco Control Initiative and Tackling Indigenous Smoking Measure were conducted as planned and, if so, the extent to which lessons learnt have informed more recent policy developments.

In terms of theories of the policy process, this case study has provided clear evidence of the role of “context/cultural rationality” and “process/political rationality” as having at least as much weight as “technical rationality” in understanding how this particular policy area evolved.

Furthermore, as this case study demonstrates, public health policy making does not always lend itself to the literal translation of rigorous research findings. More empirical case studies are needed to explore the role of other factors – such as the needs and preferences of the target population – in order to provide a clearer understanding of their relative importance in the policy process.

Author Contributions

All authors made substantial contributions to the conception and design of the work. DV conducted data acquisition and initial analysis. DV drafted the initial manuscript, and MR, RF, and SA made critical revisions for important intellectual content. All authors approved the version to be published and agree to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

Conflict of Interest Statement

Nil. Please note that, pursuant to Australia’s federal system of government, the Western Australian Department of Health (institutional affiliation for the first named author) is a separate entity to the Commonwealth Department of Health and Ageing.

Acknowledgments

The authors are grateful to Dr. Premila Webster, Prof. Jenny Lewis, Dr. Rebecca Surrender, Dr. Proochista Ariana, Prof. Sasha Shepherd, Dr. Jesse Wall, Robert Mullins, Jenny Millar, Heather Millar, and the reviewers for their valued feedback on earlier drafts.

Funding

DV received funding from the Rhodes Trust to undertake this research in partial fulfilment of the requirements for the Degree of a Doctor of Philosophy at the University of Oxford.

Footnotes

^Department of Health and Ageing. Tobacco Control and Aboriginal and Torres Strait Islander Australians. Internal Document Obtained under Freedom of Information Act 1982. (2007).
^Anonymous. Announcement of the Indigenous Tobacco Control Initiative at the Close the Gap Indigenous Health Equality Summit 20 March 2008. Internal Document Obtained under Freedom of Information Act 1982. (2008).

References

1. Marston G, Watts R. Tampering with the evidence: a critical appraisal of evidence-based policy-making. Drawing Board (2003) 3:143–63.