**TELEMEDICINE IN LOW-RESOURCE SETTINGS**

**Topic Editors Richard Wootton and Laurent Bonnardot**

#### *FRONTIERS COPYRIGHT STATEMENT*

© Copyright 2007-2014 Frontiers Media SA. All rights reserved.

All content included on this site, such as text, graphics, logos, button icons, images, video/audio clips, downloads, data compilations and software, is the property of or is licensed to Frontiers Media SA ("Frontiers") or its licensees and/or subcontractors. The copyright in the text of individual articles is the property of their respective authors, subject to a license granted to Frontiers.

The compilation of articles constituting this e-book, wherever published, as well as the compilation of all other content on this site, is the exclusive property of Frontiers. For the conditions for downloading and copying of e-books from Frontiers' website, please see the Terms for Website Use. If purchasing Frontiers e-books from other websites or sources, the conditions of the website concerned apply. **DOI** 10.3389/978-2-88919-505-32015

Images and graphics not forming part of user-contributed materials may not be downloaded or copied without permission.

Individual articles may be downloaded and reproduced in accordance with the principles of the CC-BY licence subject to any copyright or other notices. They may not be re-sold as an e-book.

As author or other contributor you grant a CC-BY licence to others to reproduce your articles, including any graphics and third-party materials supplied by you, in accordance with the Conditions for Website Use and subject to any copyright notices which you include in connection with your articles and materials.

All copyright, and all rights therein, are protected by national and international copyright laws.

The above represents a summary only. For the full conditions see the Conditions for Authors and the Conditions for Website Use.

**ISSN** 1664-8714 **ISBN** 978-2-88919-308-0 **DOI** 10.3389/978-2-88919-308-0 **ISSN** 1664-8714 **ISBN** 978-2-88919-505-3

## *ABOUT FRONTIERS*

Frontiers is more than just an open-access publisher of scholarly articles: it is a pioneering approach to the world of academia, radically improving the way scholarly research is managed. The grand vision of Frontiers is a world where all people have an equal opportunity to seek, share and generate knowledge. Frontiers provides immediate and permanent online open access to all its publications, but this alone is not enough to realize our grand goals.

## *FRONTIERS JOURNAL SERIES*

The Frontiers Journal Series is a multi-tier and interdisciplinary set of open-access, online journals, promising a paradigm shift from the current review, selection and dissemination processes in academic publishing.

All Frontiers journals are driven by researchers for researchers; therefore, they constitute a service to the scholarly community. At the same time, the Frontiers Journal Series operates on a revolutionary invention, the tiered publishing system, initially addressing specific communities of scholars, and gradually climbing up to broader public understanding, thus serving the interests of the lay society, too.

## *DEDICATION TO QUALITY*

Each Frontiers article is a landmark of the highest quality, thanks to genuinely collaborative interactions between authors and review editors, who include some of the world's best academicians. Research must be certified by peers before entering a stream of knowledge that may eventually reach the public - and shape society; therefore, Frontiers only applies the most rigorous and unbiased reviews.

Frontiers revolutionizes research publishing by freely delivering the most outstanding research, evaluated with no bias from both the academic and social point of view.

By applying the most advanced information technologies, Frontiers is catapulting scholarly publishing into a new generation.

## *WHAT ARE FRONTIERS RESEARCH TOPICS?*

Frontiers Research Topics are very popular trademarks of the Frontiers Journals Series: they are collections of at least ten articles, all centered on a particular subject. With their unique mix of varied contributions from Original Research to Review Articles, Frontiers Research Topics unify the most influential researchers, the latest key findings and historical advances in a hot research area!

Find out more on how to host your own Frontiers Research Topic or contribute to one as an author by contacting the Frontiers Editorial Office: researchtopics@frontiersin.org

## **TELEMEDICINE IN LOW-RESOURCE SETTINGS**

Topic Editors:

**Richard Wootton,** University Hospital of North Norway, Norway **Laurent Bonnardot,** Paris Descartes University, France

Telemedicine networks to support healthcare workers in resource-limited settings (often for humanitarian purposes) have evolved over the last decade or so in a largely autonomous way. Communication between them has been informal and relatively limited in scope. This situation could be improved by developing a comprehensive approach to the collection and dissemination of information.

Image copyright Laurent Bonnardot

A recent review identified seven telemedicine networks, each of which had been in operation

for at least five years and which provided store-and-forward telemedicine services to doctors in low- and middle-income countries. These networks provide clinically useful services and improved healthcare access. However, like much of telemedicine, the formal evidence for their cost-effectiveness remains weak.

Topics of current research interest therefore include the cost-effectiveness of telemedicine in resource-limited settings. Outcomes data (and methods for gathering it) such as patient quality of life following a telemedicine episode, the knowledge-gain of healthcare staff involved in telemedicine, and staff recruitment and retention in rural areas are also of interest. Finally, there is little published information about the performance of these telemedicine networks (and methods for measuring it), about how best to manage them, and about how to share resources between them.

A collection of articles reporting the current evidence supporting the use of telemedicine in resource-limited settings would build the evidence base and should provide a focus for future research. It would also serve to raise the profile of this potentially important research field.

# Table of Contents

## *05 Telemedicine in low-resource settings* Richard Wootton and Laurent Bonnardot


John D. Piette, Helen Valverde, Nicolle Marinec, Rachel Jantz, Kevin Kamis, Carlos Lazo de la Vega, Timothy Woolley and Bismarck Pinto on behalf of the University of Michigan 2013 Bolivian m-health Summer Internship

*28 The RAFT telemedicine network: lessons learnt and perspectives from a decade of educational and clinical services in low- and middle-incomes countries*

Georges Bediang, Caroline Perrin, Rafael Ruiz de Castañeda, Yannick Kamga, Alexandre Sawadogo, Cheick Oumar Bagayoko and Antoine Geissbuhler

## *34 Tele-education in South Africa*

Maurice Mars

## *44 Telemedicine networks of EHAS Foundation in Latin America*

Ignacio Prieto-Egido, Javier Simó-Reigadas, Leopoldo Liñán-Benítez, Víctor García-Giganto and Andrés Martínez-Fernández


Kokou Adambounou, Victor Adjenou, Alex P. Salam, Fabien Farin, Koffi Gilbert N'Dakena, Messanvi Gbeassor and Philippe Arbeille

*63 The Pacific Island Health Care Project*

Donald Ames Person

*68 Pediatric teleradiology in low-income settings and the areas for future research in teleradiology*

Savvas Andronikou

*73 The development of a multilingual tool for facilitating the primary-specialty care interface in low resource settings: the MSF tele-expertise system*

Laurent Bonnardot, Joanne Liu, Elizabeth Wootton, Isabel Amoros, David Olson, Sidney Wong and Richard Wootton

## *82 Teleradiology usage and user satisfaction with the telemedicine system operated by Médecins Sans Frontières*

Jarred Halton, Cara Kosack, Saskia Spijker, Elizabeth Joekes, Savvas Andronikou, Karen Chetcuti, William E. Brant, Laurent Bonnardot and Richard Wootton

## *88 Teledermatology in low-resource settings: the MSF experience with a multilingual tele-expertise platform*

Sophie Delaigue, Jean-Jacques Morand, David Olson, Richard Wootton and Laurent Bonnardot

## *97 A retrospective analysis of pediatric cases handled by the MSF tele-expertise system*

Daniel Martinez Garcia, Laurent Bonnardot, David Olson, Harriet Roggeveen, Jaap Karsten, Peter Moons, Myrto Schaefer, Joanne Liu and Richard Wootton

## *104 Assessing the quality of teleconsultations in a store-and-forward telemedicine network*

Richard Wootton, Joanne Liu and Laurent Bonnardot

*112 Assessing the quality of teleconsultations in a store-and-forward telemedicine network – long-term monitoring taking into account differences between cases*

Richard Wootton, Joanne Liu and Laurent Bonnardot

*119 Quality assurance of teleconsultations in a store-and-forward telemedicine network – obtaining patient follow-up data and user feedback*

Richard Wootton, Joanne Liu and Laurent Bonnardot

EDITORIAL published: 21 January 2015 doi: 10.3389/fpubh.2015.00003

## Telemedicine in low-resource settings

## **RichardWootton1,2\* and Laurent Bonnardot 3,4**

<sup>1</sup> Norwegian Centre for Integrated Care and Telemedicine, University Hospital of North Norway, Tromsø, Norway

<sup>2</sup> Faculty of Health Sciences, University of Tromsø, Tromsø, Norway

<sup>3</sup> Fondation Médecins Sans Frontières, Paris, France

<sup>4</sup> Department of Medical Ethics and Legal Medicine (EA 4569), Paris Descartes University, Paris, France

\*Correspondence: r\_wootton@pobox.com

**Edited and reviewed by:**

Connie J. Evashwick, George Mason University, USA

**Keywords: telemedicine, low-resource settings, telehealth, health care, editorial**

Telemedicine is a fuzzy term with several synonyms (telehealth, e-health, etc), which cover a wide range of topics, all concerning the delivery of health care at a distance. "Health care" itself is a broad concept, encompassing diagnosis and treatment of patients, education of staff, patients, and the general public, and administrative activities, such as collecting public health data, as well as research. All of these may be assisted by judicious use of telemedicine.

The main advantage of telemedicine is that it can improve access to health care, often by increasing the speed with which a specialist opinion can be obtained (e.g., tele-stroke) or by reducing the need to travel (e.g.,teledermatology); in certain disciplines,evidence has also been obtained that telemedicine is cost-effective (1). Much of the experience with telemedicine in the last 20 years has concerned its application in high-income countries. In contrast, there has been relatively little use of telemedicine in low-income countries, which is surprising in view of the difficulties of accessing health care there. In those countries where telemedicine has been trialed, it seems to have worked well and a small number of programs have provided services for periods of 10 years or more (2). These long-running telemedicine programs have mainly used store-and-forward methods, although there has been some limited use of real-time video.

The present Research Topic focuses on *Telemedicine in Low-Resource Settings,* environments where it is always a challenge to provide patients with the best level of health care. The term "low-resource settings" covers most low-income countries, and also includes regions in middle- or high-income countries where under-served populations have difficulties in accessing specialists. The Research Topic documents real, practical experience with the use of telemedicine in low-resource settings and identifies research problems of current interest. This collection of articles shows the rich diversity of applications for telemedicine. Examples come from all over the world and from a range of clinical settings and medical specialties.

Mobile phones have great potential in the delivery of health care in low-resource settings. Patterson (3) developed a mobile-phone app to enable non-doctors to diagnose episodes as epileptic. In a pilot trial with health workers in Nepal who used the app in small numbers of patients, there were no false diagnoses. This represents a potential method of empowering health workers to help the millions of people in the resource-poor world with untreated epilepsy. Ndlovu et al. (4) conducted trials with mobile-phone telemedicine in Botswana, in four medical specialties: radiology, oral medicine, dermatology, and cervical cancer screening. The benefits reported by pilot project users were sufficient to convince the government to scale up the program, which is now in progress. Both senior management support and local "ownership" of the program are thought to be important for future success. Piette et al. (5) also reported on the importance of collaborating with the local ministry of health when scaling up a mobile telemedicine application in Bolivia. All these experiences reinforce the need to develop telemedicine by scaling it up from pilot projects, to do so in collaboration with local healthcare workers (rather than trying to impose telemedicine from above) and to enlist the support of the appropriate ministry of health.

One of the longer-running examples of telemedicine used in low-resource settings is the RAFT network, which provides both educational and clinical services to centers in Africa and South America (6). The educational activities include the weekly delivery of video-lectures for continuing and postgraduate medical education. Much of this early video delivery depended on the use of satellite links, which are relatively expensive, and in recent years the RAFT program has begun to make use of lowbandwidth Internet connections. In South Africa, a tele-education network evolved from a failed government telemedicine program (7). Over 1000 h of videoconferenced lectures are delivered each year in KwaZulu-Natal, using ISDN transmission. Finally, the EHAS group has provided video-based telemedicine services in South America (8). In order to secure sufficient bandwidth for the delivery of video, they have developed long-range WiFi transmission.

An alternative method of transmitting video for telemedicine is to make use of free or low-cost web-based tools. For example, Jefee-Bahloul (9) conducted a pilot trial of telepsychiatry in Jordan using Skype, while Adambounou et al. (10) used the file transfer facilities of the LogMeIn web service for tele-ultrasound between Togo and France.

It is clear from these reports that video telemedicine is possible in low-resource environments, but it is also the case that non-realtime (store-and-forward) telemedicine is more common in these settings, not only because it is usually cheaper but also because the non-synchronous nature of the interaction between the parties makes it easier to organize. The longest-running such network is probably operated by the US military in the Pacific, which has used email and web-based communication in the Pacific Island Health Care Project since the late 1990s. As Person reports (11), teleconsultation has enabled local treatment in the Pacific islands, without necessarily requiring transfer to the major medical center on Hawaii; many of the cases were pediatric. Andronikou (12) reviewed his experience of pediatric teleradiology with three different store-and-forward programs. He concluded that teleradiology offers the potential to alleviate radiologist shortages in under-served areas, but that there are many challenges to designing an adequate process.

Médecins Sans Frontières (MSF), an organization that works mainly in low-resource settings, developed its own telemedicine tool based on the Collegium Telemedicus model (13). The aim was a system that would improve the primary-specialty care interface and allow their field doctors to obtain an expert opinion within a few hours, wherever they were located in the world. Based on a retrospective analysis and user survey, Bonnardot et al. (14) provide a general overview of the system and the user perceptions of it. The three main specialties used in the network are radiology, pediatrics, and dermatology, which were reviewed by Halton et al. (15), Delaigue et al. (16), and Martinez Garcia et al. (17), respectively.

The MSF experience, and that of others reported here, suggests that store-and-forward networks are clinically useful, sustainable, and potentially cost-effective. It is also clear that there is still lingering skepticism from some healthcare staff about the adoption of telemedicine into routine practice. Apparently, telemedicine is sometimes viewed as a threat or a competitor to conventional ways of working. Yet, telemedicine is simply another tool for assisting in the delivery of health care, and in low-resource settings there is often no other way to access the required resources.

As telemedicine matures to become a routine service in lowresource settings, it will become increasingly important to evaluate the quality of service being delivered and to demonstrate that this is being maintained. There is almost no published work about quality assurance in this context, and the present Topic contains three papers, which explore different aspects of this new area (18–20). While providing initial demonstrations of feasibility, each raises a number of questions for future research.

In summary, this e-book provides vignettes illustrating (largely successful) telemedicine projects of widely different kinds in various low-resource settings. It is worth noting that solutions that are found to overcome the huge constraints imposed by low-resource settings may also be useful in middle- or high-income countries. The common themes are that success depends on expanding from small pilot projects using a "bottom-up" approach with engagement of local health workers, yet also requires senior management and government support. The research agenda for the future requires us to document the cost-effectiveness of these programs, and as telemedicine matures, to demonstrate that quality improvement activities can be incorporated in the same way as is done in many other areas of health care. We can expect that in the future, the use of telemedicine – practising health care at a distance – will become a norm. Indeed, we expect that it will become so common as to be unremarkable, that the prefix *tele-* will disappear, and that all telemedicine work will be considered as part of usual practice.

## **REFERENCES**


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 28 November 2014; accepted: 02 January 2015; published online: 21 January 2015.*

*Citation: Wootton R and Bonnardot L (2015) Telemedicine in low-resource settings. Front. Public Health 3:3. doi: 10.3389/fpubh.2015.00003*

*This article was submitted to Public Health Education and Promotion, a section of the journal Frontiers in Public Health.*

*Copyright © 2015 Wootton and Bonnardot. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

## Telemedicine for epilepsy support in resource-poor settings

## **Victor Patterson\***

Dhulikhel Hospital, Dhulikhel, Nepal

## **Edited by:**

Richard Wootton, University Hospital of North Norway, Norway

#### **Reviewed by:**

Himanshu Negandhi, Public Health Foundation of India, India Laurent Bonnardot, Paris Descartes University, France

#### **\*Correspondence:**

Victor Patterson, 58 Lisnabreeny Road East, Belfast BT6 9SS, UK e-mail: vhp498@gmail.com

**The Problem:** Epilepsy is a common disease worldwide causing significant physical and social disability. It is one of the most treatable neurological diseases. Yet, in rural, poorer countries like much of India and Nepal, most people with epilepsy are not undergoing any treatment often because they cannot access doctors.

**Conventional Approaches:** It is being appreciated that perhaps doctors are not the solution and that enabling health workers to treat epilepsy may be better. Few details, however, have been put forward about how that might be achieved.

**Thinking Differently:** Untreated epilepsy should be considered a public health problem like HIV/AIDS, the various steps needed for treatment identified and solutions found.

**Telemedicine Approaches:** Telemedicine might contribute to two steps – diagnosis and review. A tool that enables non-doctors to diagnose episodes as epileptic has been developed as a mobile phone app and has good applicability, sensitivity, and specificity for the diagnosis.There are a number of ways in which the use of phone review or short messaging service can improve management.

**Conclusion:** Telemedicine, as part of a public health program, can potentially help the millions of people in the resource-poor world with untreated epilepsy.

**Keywords: epilepsy, untreated epilepsy, phone app, telemedicine, teleneurology, epilepsy treatment gap, developing countries**

Telemedicine is not an end in itself but a method of medical practice, which either enables the reach of medicine to be extended beyond what can be done by conventional means or enables medicine to be practised more efficiently or effectively than by conventional means (1). Too often telemedicine papers start with the technology and perhaps deal with the problem later. This paper will start with a detailed evaluation of the problem of untreated epilepsy, discuss the necessary strategies to solve it, and finish with how telemedicine might fit into those strategies in either of the roles above.

## **PROBLEM OF UNTREATED EPILEPSY**

Epilepsy is a common disease worldwide. Its prevalence is approximately 1 in 200 people but most studies from the poorer parts of the world have shown a prevalence of about twice that (2). This increase has been attributed to a combination of brain infections such as neurocysticercosis, poorer obstetric care leading to more perinatal brain disease, and more head trauma.

The effects of uncontrolled epilepsy are potentially serious. Death in epilepsy is estimated to occur with increased frequency in both rich and poor countries. There are two main causes for this: (1) accidents, in particular falls and drowning, where someone has a seizure in a vulnerable position and (2) sudden unexplained death in epilepsy (SUDEP) in which having a seizure leads to death by unexplained but possibly multifactorial causes (3). Seizures occurring when the person is in vulnerable positions can also cause disabilities ranging from burns to fractures and brain injury. Its arguably much more common effect is the social isolation which it produces. This may be in the form of family concern for the person's vulnerability and safety or by society's view of the person's disease. Societal prejudice against people with epilepsy is not new but in richer countries it has been lessened by public education emanating from patient-based charities and public health agencies. This problem is much greater in poorer societies where often epilepsy is not recognized as a medical condition at all and notions such as possession by evil spirits are more widespread. The stigma of epilepsy is a particular problem for girls with epilepsy as their marriage prospects are significantly diminished by this diagnosis (4).

Yet, epilepsy is a treatable condition. In resource-rich countries about two-thirds of people with it have their seizures completely controlled on simple medicines (5). This makes it one of the most treatable conditions not only just within neurology but also in medicine with a number needed to treat (NNT) to effect one cure of two. In rich countries, the treatment gap, the number of people not on treatment as a percentage of the total population with epilepsy, is negligible but in poorer countries it averages 75% in the many studies where it has been ascertained (2, 6). There are a number of reasons for this treatment gap: inability to pay for long-term medicines is one; lack of awareness that epilepsy is a medical disease and lack of doctors are others (6–8). The last is particularly prevalent in countries such as India and Nepal where most people live in rural areas and almost all doctors live in cities.

## **CONVENTIONAL APPROACHES**

The International League against Epilepsy (ILAE) and the World Health Organization (WHO) jointly conceived a global campaign against epilepsy in 1997. This supported a number of pilot projects in a number of different countries but its efforts have had no generalizable effect on the problem throughout the resource-poor world and it has largely fizzled out (9). Since one of the problems is a large shortage of doctors in rural areas and since there has been no obvious solution to the scale of this problem, it is being increasingly appreciated that perhaps increasing doctors is not the solution and that enabling health workers to treat epilepsy may be the way forward (2, 8, 10). Few details, however, have been put forward about how this might be achieved. A detailed proposal about how untreated epilepsy might be dealt with in India using existing doctors has been published but has not been implemented (11).

## **THINKING DIFFERENTLY – A PUBLIC HEALTH APPROACH**

Although epilepsy in general is regarded as a neurological problem, and therefore, within the remit of neurologists almost exclusively, it might be more useful to regard *untreated* epilepsy as a public health problem, and extend its remit much wider than just neurologists. This approach would actually deal with untreated epilepsy in the same way that malaria and HIV/AIDS are, even though they both are infectious diseases.

As part of this approach the individual steps for both community and individual management need to be defined and then ways of dealing with each of these steps determined, a type of project management. Once these steps are identified then solutions to them can be devised, tested, and hopefully funded. This is essentially the approach that has been used in HIV/AIDS.

## **DEFINING THE STEPS**

A summary of the possible components needed for a public health approach to untreated epilepsy is shown in **Table 1**.

Each of these steps can improve matters to some extent in isolation; for example, identifying, diagnosing, and treating untreated epilepsy resulted in 50% of people being maintained on treatment at 8 months without any review arrangements (12). However, they are likely to be much more effective when combined.

If non-doctors, particularly village health workers are to play a role in managing UE then they need to be empowered to carry out these steps, and in particular, those which are traditionally done by doctors such as diagnosis, treatment, and review. In order to see whether telemedicine will contribute to untreated epilepsy management then these steps need to be broken down into smaller steps. For the diagnosis of epilepsy these would be: first, *Is the*

#### **Table 1 | Necessary steps for untreated epilepsy**.


*episode an epileptic seizure or not?*; second, *If it is, is it primary or secondary epilepsy?*; third, *Does it need treatment or not?*

The first step is the most important as the consequences of getting it wrong are significant – either people who do not have epilepsy are diagnosed as having epilepsy and put on unnecessary and ineffective medication or people with epilepsy who might benefit from these drugs are diagnosed as not having epilepsy and so deprived of them. Surprisingly, even in rich countries, this diagnosis is made entirely from a history of the episode and not by any examination or investigation – there is a common misconception that an electroencephalogram (EEG) is necessary at this stage. Diagnosis is essentially a problem of pattern recognition and the more experienced the doctor then the more likely the diagnosis is to be correct. Even in the best hands there is still a surprisingly high rate of misdiagnosis, at least in those patients reaching a tertiary referral center where estimates of 20% are widely accepted (13, 14). It is therefore not possible for non-doctors to take on this role unless they can be provided with a tool, which distils and applies the knowledge of an experienced epilepsy doctor.

The second step,whether the epilepsy is primary,usually genetic in origin, or secondary, due to some structural brain abnormality, is important in deciding which anti-epileptic drug to prescribe. Sometimes this is clear from the clinical history, but where there is uncertainty an EEG is useful. EEGs are not widely available so a clinical algorithm of some sort will be necessary if this task is to be devolved to non-doctors and work on this task is in progress.

The third step, whether the epilepsy needs treating, is easily dealt with by a simple algorithm.

For treatment, the options will depend on local circumstances and simple algorithms to choose and start available medications are easily provided.

Review is more complex and currently requires the interaction of an experienced doctor with the patient, conventionally faceto-face. The shortage of experienced doctors in rural areas makes this approach impossible. Either empowering non-doctor health workers to consult face-to-face or using technology to enable remote consultations are possible alternatives.

## **TELEMEDICINE IN UNTREATED EPILEPSY: A USEFUL TOOL**

The above analysis suggests that a conventional system of medical care based on face-to-face consultation with a doctor will not be able to diagnose or review people with untreated epilepsy because there are simply not enough doctors. There seems, therefore, to be two distinct ways in which telemedicine solutions can help: first, by devolving care to non-medical health workers and empowering them with appropriate tools so that they can diagnose and review appropriately; second, by providing some services remotely using a combination of medical and non-medical personnel. Each of these might be useful in different situations.

#### **DIAGNOSIS**

A diagnostic tool to determine whether episodes of loss of consciousness are due to epilepsy or to other causes has been developed (15). This is based on the answers to 11 selected questions and results in a probability score of the episodes being epilepsy. It is broadly based on a Bayesian analysis of the likelihood of symptoms being associated or not associated with epilepsy. The selected questions were those with the highest likelihood ratios (16) of either being epilepsy or not epilepsy.

The probability score derived by this has been validated first in a small population from Nepal and second from three larger populations, two in Nepal and one in India (Victor Patterson et al., submitted for publication). The way the algorithm has been devised reflects the Bayesian way in which experienced doctors make a diagnosis of epilepsy – starting with a pre-test probability and then asking a series of questions, which individually increase or decrease probability until a final probability is reached. This process is carried out by doctors in an intuitive but nonnumerical way; the tool gives a precise number to the same process.

This tool has been developed as an app for a mobile phone as this has a number of advantages over alternative presentations. First, mobile phones are becoming much more prevalent throughout poorer countries and smartphones (which accommodate apps) are increasing their share of the market. Second, the information obtained can be retained easily on the phone for later use or uploaded to a webserver or emailed to another person. These functions could not be performed by, for example, a programmable calculator. The app has been developed for non-medical health workers and they used it in parts of the validation study. It should also be useful to inexperienced doctors who come across people with possible epilepsy.

In practice, the app separates most people into probability scores suggestive of epilepsy or non-epilepsy. Just as in the goldstandard of history taking there are patients in whom the diagnosis is uncertain but these are relatively few (**Figure 1**).

Subsequently, health workers in Nepal have used the app in small numbers of patients with no false diagnoses.

The second diagnostic problem, whether epilepsy is primary or secondary is important because it leads directly to choice of medication. It too should be amenable to a Bayesian approach and work on this is in progress.

## **REVIEW**

There are surprisingly few studies from anywhere in the world on using the telephone to review people with epilepsy. There is a randomized trial in progress comparing telephone with in-person review at a tertiary referral center in India (Mamta Singh, personal communication). Preliminary results from this have shown that there is no difference in breakthrough seizures between the groups but that the group reviewed by telephone had significantly fewer costs due to travel, accommodation, and lost wages. The same group has also shown that it is feasible to train nurses to review epilepsy patients in an Indian setting (17).

Telephone review is expedited by the high mobile phone ownership, which continues to increase in poorer countries. Communication technology infrastructure has improved in almost all countries of the world, however, poor. In particular, mobile phone usage has almost become universal and use of smartphones is increasing dramatically as these devices become cheaper (18). In another Indian study from 2011 showed that over half of patients with epilepsy who attended two rural epilepsy clinics were contactable by mobile phone 8 months later (12).

Text messaging using short messaging service (SMS) on mobile phones has been used as a way of continuing with epilepsy education in epilepsy patients under review (19). The authors of this study from Malaysia found that knowledge of epilepsy, medication adherence, and review attendance were all better in the group receiving SMS messages compared with a control group, which received conventional written information only.

## **FUTURE**

The epilepsy treatment gap in the resource-poor parts of the world is unlikely to be narrowed substantially without using telemedicine approaches. These telemedicine approaches have at least started though they are still at an early stage and there is some evidence that they can contribute to epilepsy care. Once these methods have been refined then they can be deployed in larger scale trials but this will require significant investment in what is effectively an orphan area of medical research. Making this happen will require innovative ideas such as the global fund for epilepsy (20) to be realized. Then the benefits to millions of people in the resource-poor world with untreated epilepsy would be considerable.

## **ACKNOWLEDGMENTS**

I would like to thank my co-members of Action Against Untreated Epilepsy – Mamta Singh, Janita Keating, Hemav Rajbhandari, Priya Jain, and Jerome Chin – for many helpful discussions and conversations.

## **REFERENCES**


**Conflict of Interest Statement:** The author declares that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 31 May 2014; accepted: 01 August 2014; published online: 21 August 2014. Citation: Patterson V (2014) Telemedicine for epilepsy support in resource-poor settings. Front. Public Health 2:120. doi: 10.3389/fpubh.2014.00120*

*This article was submitted to Public Health Education and Promotion, a section of the journal Frontiers in Public Health.*

*Copyright © 2014 Patterson. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

## Scaling up a mobile telemedicine solution in Botswana: keys to sustainability

#### **Kagiso Ndlovu<sup>1</sup>\*, Ryan Littman-Quinn1,2, Elizabeth Park <sup>1</sup> , Zambo Dikai <sup>1</sup> and Carrie L. Kovarik 1,2**

<sup>1</sup> Botswana-UPenn Partnership, Gaborone, Botswana

<sup>2</sup> Department of Dermatology, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA

#### **Edited by:**

Richard Wootton, University Hospital of North Norway, Norway

#### **Reviewed by:**

Laurent Bonnardot, Paris Descartes University, France David A. Simonyan, Centre hospitalier universitaire de Québec, Canada

#### **\*Correspondence:**

Kagiso Ndlovu, P O Box AC 157 ACH, Gaborone, Botswana e-mail: kaygndlovu@gmail.com

Effective health care delivery is significantly compromised in an environment where resources, both human and technical, are limited. Botswana's health care system is one of the many in the African continent with few specialized medical doctors, thereby posing a barrier to patients' access to health care services. In addition, the traditional landline and non-robust Information Technology (IT) network infrastructure characterized by slow bandwidth still dominates the health care system in Botswana. Upgrading of the landline IT infrastructure to meet today's health care demands is a tedious, long, and expensive process. Despite these challenges, there still lies hope in health care delivery utilizing wireless telecommunication services. Botswana has recently experienced tremendous growth in the mobile telecommunication industry coupled with an increase in the number of individually owned mobile devices. This growth inspired the Botswana-UPenn Partnership (BUP) to collaborate with local partners to explore using mobile devices as tools to improve access to specialized health care delivery. Pilot studies were conducted across four medical specialties, including radiology, oral medicine, dermatology, and cervical cancer screening. Findings from the studies became vital evidence in support of the first scale-up project of a mobile telemedicine solution in Botswana, also known as "Kgonafalo." Some technical and social challenges were encountered during the initial studies, such as malfunctioning of mobile devices, accidental damage of devices, and cultural misalignment between IT and healthcare providers.These challenges brought about lessons learnt, including a strong need for unwavering senior management support, establishment of solid local public-private partnerships, and efficient project sustainability plans. Sustainability milestones included the development and signing of a Memorandum of Understanding (MOU) between the Botswana government and a private telecommunications partner, the publication and awarding of the government tender to a local IT company, and the development and signing of a Memorandum of Agreement between the Ministry of Health Clinical Services department and the local tender winner. The initial system scale-up is scheduled to occur in 2014 and to ensure the project's sustainability, the system is aligned with the national eHealth strategy and local ownership of the project remains at the forefront (1).

**Keywords: mobile telemedicine, store-and-forward telemedicine, mHealth, public-private partnerships, national eHealth strategy, sustainability, low-resource setting**

## **INTRODUCTION**

Investing in a sustainable telemedicine solution that is relevant to the current health needs is an important milestone any nation would strive to achieve. Nonetheless, making such an investment in a resource-limited environment does not come easy as there are many challenges to overcome including, the buy-in of leadership from key stakeholders. Botswana is the first in Africa to invest in a nationwide mobile telemedicine scale-up project through its Ministry of Health (MoH). This project is in partnership with the Botswana-UPenn Partnership (BUP), the Orange Foundation of Botswana and 5AM Consultancy group. Formal discussion among key project stakeholders began after the completion of the pilot projects in 2010, and in 2012 the MoH officially agreed to scale-up the solution, formally known as "Kgonafalo" (2). The Kgonafalo mobile telemedicine system covers four medical specialties being oral health, dermatology, radiology, and cervical cancer screening, all of which involve visual inspection and follow the same workflow. The Kgonafalo project aims to improve access to specialized healthcare service delivery throughout Botswana hence improving patients' outcomes and also reducing hospital congestion in the few available referral facilities.

## **BACKGROUND AND PROBLEM STATEMENT**

Access to specialized health care continues to be a challenge across the African continent (3). This is mostly brought about by the limited number of specialized medical doctors in African countries as compared to developed continents. In order to augment the limited access to health care, many African countries have implemented innovative mobile health (mHealth) pilot project solutions (3). Most of these pilot solutions address public health problems, infectious disease epidemics and pandemics, psychosocial problems such as addiction and suicide, and other social problems such as family planning and smoking cessation, rather than access to specialty care. In addition, the majority of African countries have not yet scaled up mHealth pilot solutions to evaluate their nationwide impact, and no mobile telemedicine programs have been nationally implemented (3). Despite the development of innovative mHealth solutions, there has always been another challenge among many African health systems being that of inefficient information communication technology (ICT) infrastructure characterized by slow bandwidth (3). The recent ICT advancements revived hope for telecommunication technology as a tool to improve health care service delivery through innovative solutions such as telemedicine or telehealth. Telemedicine is the use of telecommunications technology for medical diagnostic, monitoring, and therapeutic purposes where distance and/or time separates the patient and health care provider (4).While others use the broader term telehealth to focus on all health interactions using telecommunications services, many refer to telemedicine in specific domains by using the prefix "tele"with the respective specialty such as teleradiology, teledermatology, and so on. In general, telephone only service is not considered to be true telemedicine (4). The high penetration coupled with low cost of mobile technologies places it well as the ideal telemedicine tool for low-resource settings.

The presence of mobile technology has changed many lives in Africa and nearly two-thirds (65%) of households in 23 countries in sub-Saharan Africa had at least one mobile phone in 2013, with median growth of 27% since 2008, and median annual growth of 5% (5). The 2013 report by the International Telecommunication Union (ITU), predicts that there will soon be as many mobile-cellular subscriptions as people inhabiting the planet, with the figure set to nudge past the seven billion mark early in 2014 (5). The report further states that more than half of all mobile subscriptions are now in Asia, which remains the powerhouse of market growth and by the end of 2013 overall mobile penetration rates will have reached 96% globally, 128% in the developed world, and 89% in developing countries (5). Given the fast growth of mobile technologies, security is a major concern particularly when it comes to health information. Network security standards and protocols for mobile telecommunications have advanced over the past years (6). Internet data security protocols such as the secured version of hyper-text transfer protocol (HTTPS), the Secure Socket Layer (SSL) encryption standard, and the Advanced Encryption Standard (AES) are some of the recommended security measures for the transmission of sensitive data over mobile networks (6). The Health Insurance Portability and Accountability Act (HIPAA) continues to offer encryption standards on Patient Health Information (PHI). The HIPAA demands that all HIPAA covered businesses prevent unauthorized access to "Protected Health Information." PHI includes patients' names, addresses, and all information pertaining to the patients' health and payment records (7).

Some uses of telemedicine include facilitation of access to health care in underserved rural communities especially for specialty care, closer monitoring of patients in their homes in urban or rural settings especially frail elderly or those with chronic disease and "outsourcing" of non-direct care services such as radiology and pathology.

Telemedicine can be classified according to the following categories; office or hospital-based telemedicine, which is an interactive substitute for face-to-face encounter utilizing live video conference with a medical doctor (4). The other type is what is commonly known as the store-and-forward telemedicine, which is purely asynchronous, that is, non-real-time encounter (4). The asynchronous type of telemedicine is more suitable for lowresource settings because of the slow bandwidth challenge. The other telemedicine category is the home-based telemedicine which entails monitoring of patient at their homes or in a nursing care facility (4).

In Botswana, the doctor to patient ratio is at 3.1 per 10,000 and this has resulted in an over-burden of the few specialist doctors available (8). Patients travel long distances hence taking longer times before receiving specialized attention resulting in complex cases and sometimes loss of lives. About 84% of the population in Botswana lives within a 5-km radius and about 95% live within an 8-km radius from a health facility (8).

In 2009, the Botswana MoH introduced the Integrated Patient Management System (IPMS) aimed at improving daily processes in healthfacilities and also leading to the creation of a single patient health record which is accessible across all health facilities. The full benefits of such an IPMS in Botswana have been hindered by the traditional landline, non-robust ICT infrastructure. Botswana is mentioned to have 87% of households with at least one mobile phone in 2013 and with 3% average annual growth since 2008 (5). Also a significant improvement has been recorded on Botswana's mobile telecommunication networks such as EDGE, GPRS, 2G, 3G, and the most recent 4G connections. The ICT challenges faced by the MoH in Botswana have inspired innovative ways of offering health care service including a telemedicine solution utilizing mobile devices.

## **PILOT PROJECTS**

Over the past 4 years, BUP has piloted four mobile telemedicine projects in the specialties of women's health (cervical cancer screening utilizing visual inspection with acetic acid), radiology, oral medicine, and dermatology (1). Mobile telemedicine has been implemented in 15 locations in Botswana, 27 clinicians have been trained, and 696 cases have been successfully managed (see **Table 1**).

## **PILOT PROJECT WORKFLOW**

Healthcare workers were provided with smart phones equipped with a built-in camera and data-enabled subscriber identity module (SIM) cards donated by the Orange Foundation of Botswana. The organizational structure of each mobile telemedicine project includes an in-country medical specialist, an international specialist (the only position that is not held by a MoH employe), a national specialty manager, a referral site coordinator, and the referring health care workers.Allfour mobile telemedicine projects used the same model in which the healthcare worker collects pertinent clinical history and associated images pertaining to a complex



patient case utilizing a smartphone with a telemedicine application (1). Before gathering patient information, a consent form is filled with the patient to allow them to indicate whether they would like to participate in the study or not. The collected history and images are then sent via mobile phones directly by the referring health care worker to an in-country remote specialist for consultation. The specialist uses the information to diagnose the illness and recommend an appropriate course of treatment. The in-country specialist also has the option of forwarding the case to an international specialist for further input and collaboration. The national specialty manager is the uniting force for each program, working to train the health care workers on site, as well as coordinating all parties involved in the scaling up and sustainability of their program.

Some of the benefits identified from the pilot studies include avoiding lengthy waiting times for hospital treatment, cost saving in petrol during referrals, reduction in patient's long distance travels, and quick medical attention hence improving patient outcomes (1). The pilots also indicate that MoH nurses and other health care workers involved in the scale-up project will gain more knowledge as they regularly interact with the telemedicine solution through uploading of cases and viewing specialists' responses over time.

In addition, a business research study was conducted by five University of Botswana MBA students in collaboration with University of Pennsylvania Wharton MBA students during the pilot. The purpose of the study was to assess the cost-benefit analysis of a mobile telemedicine solution across all the four medical specialties in Botswana and also propose a sustainable business model. After conducting an economic research analysis, the MBA students found out that the mHealth intervention provided cost savings per patient visit over the first year of implementation.

## **PILOT PROJECT'S MOBILE DEVICES**

The initial pilot studies utilized the Sony Ericsson C905 to take images and collect patient information on proprietary software from a US-based software company, ClickDiagnostics. All patients' data were sent to a central database server where specialists accessed them prior to making a diagnosis. Refer Table S1 in the Supplementary Material for a summary of the technical specifications for the Sony Ericsson C905 mobile device.

During 2011, the pilot technology transitioned to use the T-Mobile myTouch 3G Slide Android phones, which allowed partners to use the Open Data Kit (ODK) open-source software supported by a local IT group called PING. Refer Table S2 in the Supplementary Material for technical specifications for the T-Mobile myTouch 3G slide mobile device.

## **PILOT PROJECT'S MOBILE APPLICATION**

Customized software from ClickDiagnostics was initially utilized to capture patients' cases during the pilot project phases. The cases, each consisting of at least thirty unique medical data fields and one to multiple photographs, were sent from the phones over a data-enabled mobile network and stored on a secure database over the internet. The ClickDiagnostics software was later replaced by an open-source mobile application, the ODK, which also captured pertinent patients'information on mobile smartphones. The choice of the ODK was influenced by the fact that it is free software available on the Internet and has been tried and tested by a community of open-source developers, and it is easy to upgrade without losing data when newer versions are installed. The other advantage with ODK was its flexibility when creating and customizing forms for data entry. The ODK forms also captured patients' data, which were later saved for transfer to an online database on a cloud based infrastructure.

## **PILOT PROJECTS OBJECTIVES AND EVALUATIONS**

Each of the pilot projects focused on specific objectives and was evaluated based on the respective objectives as shown below:


## **SCALE-UP PROJECT**

The benefits reported by pilot project users were vital to the adoption of Kgonafalo system into the MoH's long-term strategy (1). The business study's promising results coupled with observed improved patient outcomes as a result of the pilot projects, contributed to the MoH support and setting aside funds to support Kgonafalo.

The MoH ultimately committed a budget from the Oral Health Clinical Services recurrent budget to support the Scale-Up of Kgonafalo for oral medicine, dermatology, radiology, and cervical cancer screening.

## **TENDERING PROCESS**

In 2013, the MoH Clinical Services published the Invitation to Tender (ITT) documents for supporting the project scale-up (14, 15). This was a public invitation for both local and international companies to submit proposals. BUP facilitated a bidders' meeting to clarify the needs of the system and answer questions by local IT businesses. Six local companies some of which were in collaboration with international firms submitted their tender proposals.

The three most competitive proposals were evaluated using a template developed by the MoH projects unit. A team of three members conducted the tender evaluation process. The Procurement Department advised on financial evaluations of the proposals and the project's main contact person came from the MoH Clinical Services Department. The successful proposal evaluation stage led to the tender selection and award process toward the most competitive bidder.

The tender was awarded in December 2013 to 5AM Holdings. The process of developing and publishing the ITT, then evaluating the proposals and choosing the winner occurred over the course of 1 year and 5 months.

Talks with the MoH IT management in January 2014 resulted in an agreement to have the Kgonafalo system hosted on servers at the MoH headquarters. This is one way of ensuring continual system support and ownership of the project by the main stakeholder.

## **PUBLIC-PRIVATE PARTNERSHIPS**

An MOU was signed in 2013, between the Orange Foundation of Botswana and the MoH. The MOU detailed collaboration roles for each partner in support of the project for the next 3 years. The MOU mentions 19 locations across Botswana as initial project scale-up sites. The Orange Foundation of Botswana committed to donating all mobile devices together with customized SIM cards loaded with data bundles for the project. The process of developing and finalizing the contents of the MOU with respective technical and legal units occurred over the course of 1 year and 7 months.

After the signing of the MOU, The Orange Foundation of Botswana through the support of BUP, identified test mobile devices to be evaluated by all Kgonafalo system users. A successful mobile device evaluation and test session was conducted by specialty managers for the project and user feedback was documented and shared with the Orange Foundation of Botswana.

## **SCALE-UP PROJECT'S TECHNICAL DETAILS**

Based on feedback from the pilot studies, the BUP provided recommendations to Orange Botswana concerning the specifications requirements for mobile devices for each specialty area. The Orange Foundation of Botswana identified the Alcatel One Touch Idol Ultra 6033X smartphone, in consultation with the BUP. Refer Table S3 in the Supplementary Material for a list of technical details for the selected Alcatel smartphone following test session across the four specialist areas (see **Table 2** for feedback from the test sessions).

In addition to donating mobile devices, the Orange Foundation of Botswana also donated customized SIM data bundles to support connectivity of mobile devices with the system central database. Confidentiality of sensitive patient information is an integral component of the Kgonafalo system. None of the patient's data is compromised or made accessible through the Orange Botswana servers and network. This has been made possible through the adoption of security protocols such as the SSL encryption, AES, and HIPAA encryption standards on both the mobile application and the enterprise system.

The Orange Foundation of Botswana released an updated network coverage map showing their nationwide 3G, EDGE, and GPRS coverage (16). The Network Coverage Map from the Orange Foundation of Botswana alongside the Kgonafalo expansion locations map are included as part of the Supplementary Material Figure S1.

## **SUSTAINABILITY STEPS**

Sustainability is a major goal of the Kgonafalo project. Sustainability here refers to the life-long operation of Kgonafalo mobile telemedicine solution. Close working relations among the project stakeholders contributed to numerous processes, documents, and presentations being developed to ensure that expansion of the project are conducted responsibly and with a focus on local ownership and drive. In order to achieve this goal, the project partners have developed operational and economical milestones as shown below.

#### **Operational milestones**

• Information technology (development, maintenance and support) supporting the mHealth system is supported by local IT groups and utilizes open-source software.




## **Financial milestones**


Efforts have been focused on four main components of sustainability: local customization, local ownership, public-private partnerships (PPP), and the IT provider business model. Empowerment of local ownership and drive are consistently encouraged throughout the scale-up project. A significant amount of planning and major decision making is left with specialty managers and site coordinators who have embraced the projects and pursued initiatives to improve them. The specialty managers organize and run training workshops and sensitization programs for local healthcare workers, allowing them to raise awareness of telemedicine and mHealth. Specialty managers and site coordinators developed the idea of telemedicine awareness posters to hang in all the field sites. Refer Figure S2 in the Supplementary Material for both Setswana and English versions of the mobile oral telemedicine awareness posters. In another step toward local ownership, an official mHealth help desk is currently being developed for all mHealth phone users, which will feed into the existing MoH help desk system. One major sustainable component of Kgonafalo is the fact that it is institutionalized into existing government health programs that can receive budget attention.

## **FUTURE DIRECTION**

Looking ahead, the MoH is interested in expanding the Kgonafalo project to cover more medical specialties and subsequently cover the whole country. The system has already been built to interface with the MoH electronic medical record system, IPMS; however, there could be more opportunities to interface with more IPMS modules. The data and images captured in Kgonafalo could be used for educational purposes as well as to feed into future decision support systems at the MoH. During the scale up, separate

funding will be set aside by the MoH for maintenance and operational support services once the service-level agreement with the local IT group comes to an end.

## **CONCLUSION**

BUP, MoH, and the Orange Foundation of Botswana have successfully embarked on a country-wide mobile telemedicine project following successful pilot studies. When not properly planned for and mitigated, scaling up such a project can lead to major setbacks. Kgonafalo has received substantial support from all local stakeholders. The project has attracted local ownership through the PPP engagement. Since the tender award stage, project planning and management by the local partners has been efficient and timelines have been met. The success of the pilot projects and recent scale-up achievements shows that if properly managed, the Kgonafalo solution could be of significant value to Botswana's healthcare system. The project's major benefits include substantial reduction of unnecessary referral costs and improved patient outcomes. A number of challenges have been encountered but did not divert stakeholders' focus from the project's goals. Recent project developments include the procurement of 42 mobile devices by the Orange Foundation of Botswana and the development of beta version of the mobile telemedicine solution by the engaged local IT group. The success of the scale-up project is largely influenced by senior managements' support and commitment to the project. Ongoing strong local partnerships and support by specialty managers are equally important for the scale-up project to be a success.

## **SUPPLEMENTARY MATERIAL**

The Supplementary Material for this article can be found online at http://www.frontiersin.org/Journal/10.3389/fpubh.2014. 00275/abstract

## **REFERENCES**


16. Orange Foundation. *Network Coverage Map*. (2014). Available from: http: //www.orange.co.bw/mobile/coverage.php

**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 05 June 2014; accepted: 25 November 2014; published online: 11 December 2014.*

*Citation: Ndlovu K, Littman-Quinn R, Park E, Dikai Z and Kovarik CL (2014) Scaling up a mobile telemedicine solution in Botswana: keys to sustainability. Front. Public Health 2:275. doi: 10.3389/fpubh.2014.00275*

*This article was submitted to Public Health Education and Promotion, a section of the journal Frontiers in Public Health.*

*Copyright © 2014 Ndlovu, Littman-Quinn, Park, Dikai and Kovarik. This is an openaccess article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

## Establishing an independent mobile health program for chronic disease self-management support in Bolivia

#### **John D. Piette1,2,3\*, Helen Valverde<sup>4</sup> , Nicolle Marinec 1,2,3, Rachel Jantz <sup>2</sup> , Kevin Kamis <sup>2</sup> , Carlos Lazo de la Vega<sup>5</sup> ,TimothyWoolley <sup>6</sup> and Bismarck Pinto<sup>5</sup> on behalf of the University of Michigan 2013 Bolivian m-health Summer Internship**

<sup>1</sup> Center for Clinical Management Research, VA Ann Arbor Healthcare System, Ann Arbor, MI, USA

<sup>4</sup> Servicio Departamental de Salud, La Paz, Bolivia


## **Edited by:**

Richard Wootton, University Hospital of North Norway, Norway

#### **Reviewed by:**

Chandra Y. Osborn, Vanderbilt University, USA Perry Foley, Duke University, USA

#### **\*Correspondence:**

John D. Piette, Center for Managing Chronic Disease, University of Michigan School of Public Health, 1415 Washington Heights, Ann Arbor, MI 48109, USA e-mail: jpiette@umich.edu

**Background:** Mobile health (m-health) work in low- and middle-income countries (LMICs) mainly consists of small pilot programs with an unclear path to scaling and dissemination. We describe the deployment and testing of an m-health platform for non-communicable disease (NCD) self-management support in Bolivia.

**Methods:**Three hundred sixty-four primary care patients in La Paz with diabetes or hypertension completed surveys about their use of mobile phones, health and access to care. One hundred sixty-five of those patients then participated in a 12-week demonstration of automated telephone monitoring and self-management support. Weekly interactive voice response (IVR) calls were made from a platform established at a university in La Paz, under the direction of the regional health ministry.

**Results:**Thirty-seven percent of survey respondents spoke indigenous languages at home and 38% had six or fewer years of education. Eighty-two percent had a mobile phone, 45% used text messaging with a standard phone, and 9% had a smartphone. Smartphones were least common among patients who were older, spoke indigenous languages, or had less education. IVR program participants completed 1007 self-management support calls with an overall response rate of 51%. IVR call completion was lower among older adults, but was not related to patients' ethnicity, health status, or healthcare access. IVR health and self-care reports were consistent with information reported during in-person baseline interviews. Patients' likelihood of reporting excellent, very good, or good health (versus fair or poor health) via IVR increased during program participation and was associated with better medication adherence. Patients completing follow-up interviews were satisfied with the program, with 19/20 (95%) reporting that they would recommend it to a friend.

**Conclusion:** By collaborating with LMICs, m-health programs can be transferred from higher-resource centers to LMICs and implemented in ways that improve access to self-management support among people with NCDs.

**Keywords: mobile health, disease management, chronic illness, vulnerable populations, Latin America**

## **INTRODUCTION**

## **M-HEALTH IN LOW AND MIDDLE-INCOME COUNTRIES**

Most people die from non-communicable diseases (NCDs), 80% of which occur in low- and middle-income countries (LMICs) where mortality rates are twice those of industrialized nations (1– 4). The years of life lost to diabetes increased by 29% from 2000 to 2010 and the global prevalence of diabetes will be 439 million in 2030 (5, 6). More than one in four adults worldwide has hypertension, with two-thirds living in LMICs (7). In Latin America, more than 100 million adults are hypertensive, with rates among the highest in the world (7). Hypertension is the main contributor to the disease burden globally and the leading cause of cardiovascular diseases (1, 2, 8). Most adults with hypertension and other NCDs rely on primary care for disease management. However,LMICs still have weak primary care systems that lack capacity and resources to help patients effectively treat these conditions.

In the face of these resource constraints, models for developing mobile health (m-health) tools to improve patient monitoring and self-care support are especially important. Mobile phones are ubiquitous in LMICs (9–12), and because m-health services have low marginal costs and high availability, they have the potential to reach large numbers of patients between in-person clinical encounters. Many studies have demonstrated that m-health services including short-message service (SMS or text messaging) and

<sup>2</sup> University of Michigan School of Public Health, Ann Arbor, MI, USA

<sup>3</sup> University of Michigan School of Medicine, Ann Arbor, MI, USA

interactive voice response (IVR) calls can improve NCD self-care support. Randomized trials suggest that m-health interventions can improve outcomes of chronic illness care in LMICs as well as a range of important self-care behaviors (13–17). Studies in Africa have demonstrated that SMS can improve patients' medication adherence (18, 19), while SMS messages to health workers can improve the quality of disease management (16). SMS studies in other countries also have had promising results (20–22). Monitoring and self-care support using IVR can complement SMS and provide a scalable solution to: collect complex clinical information; deliver more complex self-care education messages; and communicate with patients who have limited literacy, vision, or dexterity for texting. However, the challenge that has frustrated the field of m-health has been termed "pilot-itis," i.e., the repeated conduct of small, short-term, and non-sustainable pilot programs without a clear model for disseminating approaches from expert centers or scaling them in order to address the needs of large populations.

We have developed potentially sustainable models for mhealth self-management support for vulnerable populations in the US and Latin America (23–30). In 2010, we collaborated with community leaders in Honduras to evaluate the feasibility of delivering m-health self-management support to patients with diabetes in that country via IVR and a cloud-computing approach (31). Patients received weekly monitoring and selfmanagement support calls, during which they reported clinical information and received tailored self-management education. Based on those reports, clinicians received automated notifications via email identifying patients needing additional assistance. At follow-up, participants reported high levels of intervention satisfaction and improvements to self-care behaviors, and glycemic control improved significantly. In 2011, we conducted a randomized trial evaluating the efficacy of m-health hypertension management among patients in Honduras and Mexico (32). Intervention patients' systolic blood pressures improved and patients reported fewer depressive symptoms, fewer medication problems, better general health, and greater satisfaction with care (32).

## **DEVELOPING M-HEALTH PROGRAMS FOR NCD MANAGEMENT IN BOLIVIA**

Bolivia is one of the most economically challenged countries in the Western Hemisphere. It has a population of 10.5 million with 51% living at or below the poverty line (33). Thirty-five percent of Bolivians live in rural areas. Bolivia's health system includes both public and private providers of "western"medicine, and traditional medical providers serve 10% of the population (34).

Since passage of national reforms in 2007, all Bolivians have a legal right to access health care. Publically provided services are available in some areas for specific population subgroups, such as the elderly, pregnant women, and infants. However, economic, geographic, cultural, and social barriers limit access to health care for 77% of the population (35). The majority of the employed population (69%) works in the informal labor market, and thus is not eligible for most public or private health insurance programs (34). In 2009, 57% of all Bolivians were without any health insurance coverage (36) and only 12% are able to pay for private health insurance (36). Even when insurance is available, supply and infrastructure constraints limit healthcare access (36, 37). Bolivia

has 1.2 doctors and 2.1 nurses per 1000 inhabitants (36), and of the 2875 primary care centers, 45% do not have a physician (2007 data) (35).

Since 2012, we have worked in collaboration with the Institute of Applied Engineering at la Universidad Católica Boliviana in La Paz to package the IVR platform, developed at the University of Michigan, for transfer and installation as an independent program in La Paz. Beginning in 2013, we initiated a multi-year project to better understand the ways in which m-health interventions could assist Bolivian health officials and clinicians in addressing the needs of the population. Here, we describe the results of a survey designed to provide information about the potential reach and scalability of m-health services for NCD self-care support in Bolivia as well as the results of initial testing of the IVR platform among patients with diabetes and/or hypertension.

## **MATERIALS AND METHODS**

## **ORGANIZATIONAL PARTNERS**

The primary organizational partners for this study were: the program on Quality Improvement for Complex Chronic Conditions at the University of Michigan (UM); the NCD epidemiology unit in el Servicio Departamental de Salud (SEDES or Departmental Health Services) in La Paz, Bolivia, and la Universidad Católica Boliviana (UCB or Bolivian Catholic University), also in La Paz. Within UCB, the project was jointly directed by the Institute for Research on Behavioral Science and the Institute for Applied Engineering, both working under the direction of the Office of the Rector. Two other universities in La Paz – la Universidad Mayor de San Andrés and la Universidad Pública de El Alto – also played important roles as facilitators and collaborators in survey data collection.

## **CROSS-SECTIONAL QUANTITATIVE SURVEY**

In order to understand NCD patients' needs for m-health support and their access to mobile phones, we conducted a face-to-face cross-sectional survey between June and August 2013. Under the direction of SEDES, potential participants were approached at the time of visits to: primary care clinics in two public hospitals and one private hospital in La Paz, one public hospital in El Alto, and two hospital-sponsored health fairs. Surveys were administered by a team of UM graduate students fluent in Spanish, working in collaboration with Bolivian clinicians and medical students. Native Spanish speakers reviewed and tested the survey prior to patient recruitment. The study was approved by both UM and UCB ethics committees and all participants provided written informed consent.

A total of 1144 patients aged 18 years or older completed the initial survey module. This module captured information about respondents' age, household income, occupation, educational attainment, marital status, reasons for their visit to the clinic or health fair, NCD diagnoses, medication use, social support, and access and utilization of mobile phones. A total of 664 participants also completed a second, more extensive survey module. Patients were eligible for this second module if they: reported having one or more diagnosed chronic conditions, had systolic blood pressures indicating potential hypertension (systolic pressures >140 mmHg), or scored positive on the Patient Health Questionnaire (PHQ) two-item depression screener (38). The more extensive survey included questions about respondents': access and utilization of health care services, alcohol use [as measured by the Alcohol Use Disorders Identification Test or AUDIT (39)], tobacco use, NCD self-care, blood pressure self-monitoring (for hypertension patients), glucose self-monitoring (for diabetes patients), perceptions about the quality of health services [as measured by the Patient Assessment of Chronic Illness Care or PACIC scale (40)], depressive symptoms [as measured by the PHQ-8 (41)], ethnicity, and language spoken at home. Patients who completed the survey were given a small gift that included toiletries such as toothbrushes, toothpaste, and lip balm. The data presented here are limited to the subsample of 364 patients who completed both survey modules and were potentially eligible for the IVR pilot study, i.e., patients who reported either a diabetes or hypertension diagnosis, or who had a systolic blood pressure at the time of the survey of >140 mmHg.

## **IVR TRIAL**

One hundred sixty-seven patients with diabetes and/or hypertension were approached for enrollment in the trial of the IVR platform. Two patients were unable to complete the enrollment because of time constraints, and the remaining 165 were enrolled. Technical details about the IVR platform, including the hardware and software requirements, are available from the authors on request. In addition to weekly IVR calls as described below, patients with hypertension received a home blood pressure monitor and were trained in its use, including how to keep a daily record of their systolic blood pressure values. All participants received 12 weekly IVR calls to their mobile or landline telephones, with multiple call attempts at times the patient indicated were convenient. Calls emanated from the m-health platform established at UCB and the system used a GSM gateway and SIM cards to make calls directly to local mobile phone towers. The IVR script used a tree-structured algorithm to assess patients' self-management behaviors, perceived health status, and symptoms. The call script was initially developed in English with input from experts in NCD management, primary care, and m-health. The script was then professionally translated into Spanish and reviewed by Bolivian primary care providers for cultural and linguistic appropriateness. Quality assurance testing was performed on all IVR programming, and website interface prototypes were evaluated by clinicians and staff in each location using free exploration and performance tests to ensure appropriate navigation and system utilization (42).

The content and structure of the IVR interactions in both English and Spanish are available as Supplementary Material. In brief, during each call, patients reported information about their perceived health status (excellent, very good, good, fair, or poor); changes in their health compared to the prior week; diabetes and antihypertensive medication adherence; whether the patient had enough medication to last 2 weeks; (for hypertensive patients) whether they or someone else was regularly checking their blood pressure at home; (if checking) whether they regularly had high systolic blood pressure readings (i.e., >130 mmHg if diabetic or >140 mmHg if non-diabetic), whether they regularly had systolic blood pressure readings <100 mmHg, and whether they were

avoiding foods high in salt. Based on patients' responses, they were given tailored self-management education and, as needed, were advised to contact their doctor. For example, if a patient with both diabetes and hypertension reported regularly experiencing systolic blood pressures >130 mmHg, he or she was told (in Spanish) the following:

Your blood pressure may be too high. Even blood pressure that is just a little higher than normal can worsen the complications of diabetes. If you cut back on your salt intake, you may be able to get your blood pressure level down. However, many people need to adjust their medication to bring their blood pressure in line. It is important that you make an appointment with your doctor soon so that you can tell your doctor that your blood pressure has been running higher than normal.

At each participating clinic, we identified a primary physician who served as the point of contact for receiving and acting on structured email notifications generated automatically based on patients' IVR health and self-care reports. Notifications included the patient's full name, and the time and date that the patient reported the information generating the alert. For patients with diabetes, clinician alerts were generated if the patient reported rarely or never taking their antihyperglycemic medication as prescribed or (if also hypertensive) the patient reported systolic blood pressure readings >130 or <100 mmHg on two or more days in the prior week. For patients with hypertension, clinical alerts were generated if the patient reported rarely or never taking their antihypertensive medication as prescribed or reported regularly experiencing systolic blood pressures >140 or <100 mmHg on two or more days in the prior week.

## **QUALITATIVE FOLLOW-UP INTERVIEWS**

We identified a purposive sample of 20 patients who participated in the IVR program for follow-up semi-structured telephone interviews. Patients were contacted via their mobile phone and all interviews were conducted by bilingual research associates. Patients' responses were documented using a semi-structured form. The interviews were not audio-recorded. We sampled patients from each recruitment site to represent diverse experiences, including patients with varying numbers of completed IVR calls, both genders, and a mixture of patients with diabetes and hypertension. Questions addressed themes such as patients' overall satisfaction with the program, whether or not the patient felt that the program helped their chronic disease self-management, and whether the patient would recommend the program to a friend or relative if that person also were diagnosed with one of the target health conditions. Respondents were also asked, "If the program were available in your clinic, would you use it again?" Out of the 37 patients initially identified, 20 completed telephone interviews, 15 could not be contacted by phone, and 2 declined participation.

## **ANALYSIS**

Analyses of the quantitative survey data focused on the bivariate relationships between patients' sociodemographic characteristics (age, gender, ethnicity, education, and need for health information), clinical characteristics (self-reported general health status and number of reported chronic diseases), and health care access; and, their access and use of m-health technology. As a proxy for indigenous ethnicity, we identified patients who reported speaking an indigenous language (i.e., a language other than Spanish, such as Aymara) in the home. High health information need was defined as self-reported illiteracy or "frequently" or "always" needing help understanding health information, as defined by a widely used screener (43). Patients with healthcare access problems were defined as those reporting that cost kept them from going to the clinic or hospital in the past year (a measure of financial access) and patients who had longer travel times to clinic (a measure of geographical access). Patients' access and use of m-health technology was characterized using a four-level hierarchical variable identifying patients without a mobile phone; patients who had a standard mobile phone but who were unable to text; patients with a standard mobile phone who used texting; and patients with a smartphone.

The primary outcome for the IVR demonstration was patient engagement defined as "call completion." We examined engagement using a dataset including one record for each week in which a contact was attempted (i.e., approximately 12 weeks per participant). We created a merged dataset linking each call week of experience to the patient's baseline survey reports and we used logistic regression to identify the patient characteristics associated with call completion. In order to ensure that the system did not automatically re-call patients who hung up early, calls were considered completed after the patient responded to the first general health question. In practice, patients who accepted an IVR call almost always stayed connected until the end of the call. Predictors of call completion for the multivariate model represented each of the patient characteristics shown in **Table 1**, and we used cubic splines and a graphical display to describe the variation in IVR call engagement across age groups controlling for covariates. To evaluate the validity of patients' IVR clinical reports, we examined the correlation between: (1) IVR health and self-care reports and (2) patients' perceived health status and medication adherence reported at baseline. Medication adherence at baseline was measured using a validated adherence scale (44). Finally, we used multivariate logistic regression (controlling for patient characteristics shown in **Table 1**) and patients' IVR reports of perceived health status to characterize changes in health status over the course of patients' 12-week participation in the program. Specifically, we constructed a model estimating the probability that the patient would report excellent, very good, or good health during an IVR call (versus fair or poor health), with call week being the predictor of interest and controlling for baseline patient characteristics. To shed light on observed changes in patients' IVR-reported perceived health over time, we examined the correlation between IVR health status reports and patients' likelihood of reporting taking their medication as prescribed during the same call. All analyses of call week-level data were conducted taking into account the clustering of calls within patient.

## **RESULTS**

## **CROSS-SECTIONAL QUANTITATIVE SURVEY**

A total of 1114 primary care patients completed the initial module of the quantitative survey of their demographic characteristics, diagnoses, and mobile phone use. Of these, 664 patients with chronic illnesses completed the more extensive survey of their self-care and service use, including 364 respondents who either reported a diagnosis of diabetes or hypertension, or who had a resting systolic blood pressure reading of 140 mmHg or higher (**Table 1**). Twenty-six percent had diabetes but not hypertension, 45% had both conditions, and the remaining 29% had hypertension but not diabetes. Patients represented a broad distribution of ages, and most respondents (59%) were women. A total of 37% reported speaking an indigenous language in the home and 38% had no more than 6 years of formal education. Most patients (70%) reported fair or poor health status and 43% reported that they avoided seeking care at least once in the prior year due to the cost. While 40% of patients lived <30 min from their source of primary care, 24% traveled more than an hour, with many of those patients coming from rural areas outside of La Paz.

Overall, 18% of respondents with diabetes or hypertension had no mobile phone, 29% had a standard mobile phone but did not text, 45% had a standard mobile phone and used texting, and only 9% had a smartphone. Patients' sociodemographic characteristics, health status, and healthcare access were related to their level of engagement with mobile phone technology. For example, while only 14% of respondents between 18 and 29 years of age had no phone, this was true for 34% of respondents over age 65 years. In contrast, 21% of respondents aged 18–29 years had a smartphone – more than five times as many as patients who were aged 65 years or higher (4%). Smartphone access was less common among patients who spoke indigenous languages in the home, had less education, or had greater health information needs.

## **IVR TRIAL**

A total of 165 patients from the participating primary care centers and health fairs participated in the trial of the Bolivian IVR platform. Twenty-three percent of patients had diabetes without hypertension, 48% had hypertension without diabetes, and the remaining 29% of patients had both conditions. Compared to patients who did not participate, IVR study participants were significantly older on average (60 versus 49 years, *P* < 0.001) but were similar in terms of years of education, gender, and ethnicity.

Interactive voice response study participants completed 1007 weekly IVR self-management support calls out of 1995 weekly calls attempted, for an overall completion rate of 51%. Thirty percent of patients completed more than nine IVR calls and 15% completed calls for 11 or 12 weeks in which one was attempted. Nineteen percent of participants were never successfully reached. In multivariate analysis, the probability of completing an IVR call was not related to patients' gender, ethnicity, health status, or self-reported healthcare access. Call completion was significantly associated with patients' level of educational attainment (*P* = 0.03). Compared to patients with not more than 6 years of schooling, the odds of call completion was 2.4 times as high (95% CI: 1.2,4.6) among users with>12 years of education and was marginally higher among users with 7–12 years (adjusted odds ratio: 1.4; CI: 0.8, 2.6). The odds of completing an IVR call were 3.0 times higher among patients with hypertension (with or without diabetes) when compared to patients with diabetes alone (*P* < 0.001), possibly due to the fact that hypertension patients received a home


**Table 1 | Characteristics of primary care patients with diabetes and/or hypertension in La Paz, Bolivia by level of engagement with mobile technology**.

Cell entries are row percents, except the N (%) column, which includes marginal/column percents. N's in some cells represent a small amount of missing data on that survey item.

<sup>a</sup>High information needs defined as being illiterate, frequently or always needing someone to help read papers from the health center, or frequently or always having problems understanding written medical instructions.

b Includes patients' report of a physician diagnosis of: diabetes, depression, cancer, hypertension, arthritis, chronic back pain, cardiovascular disease (heart attack, blocked artery, or stroke), and pulmonary disease (emphysema, COPD, or asthma).

<sup>c</sup>Self-reported physician diagnosis or measured systolic blood pressure >140 mmHg if hypertensive, or >130 mmHg if diabetic OR hypertensive and diabetic.

d In the past year, did cost ever keep you from going to a clinic or hospital?

<sup>e</sup>Column percent.

<sup>f</sup>Level 0: does not own cell phone; Level 1: owns personal cell phone but unable to text. The cell phone is not a smart phone or is shared; Level 2: owns a cell phone and able to text. Cell phone is not a smart phone; Level 3: owns a smart phone.

blood pressure monitor in addition to the IVR calls. Call completion was roughly constant across age groups until approximately age 65 years, and then declined precipitously, see **Figure 1**.

Patients' responses during IVR assessments were associated in expected ways with their health and self-care behaviors reported at baseline, see **Figure 2**. For example, 92% of patients who reported poor health at baseline reported fair or poor health at least once during an IVR assessment, compared to 71% of patients reporting fair health at baseline, and 45% of patients reporting good, very good, or excellent health at baseline (*P* < 0.001). Patients were also more likely to report problems taking their medication as prescribed during their IVR calls if they had worse medication adherence scores at baseline, reported using the Morisky measure (44).

In multivariate modeling, patients were 3.3 times as likely to report fair or poor health during an IVR call if they reported fair or poor perceived health status during their baseline survey (*P* < 0.0001). They were also more likely to report fair or poor health via IVR if they had fewer years of education, and were 2.2 times as likely to report fair or poor health via IVR if they had greater baseline-reported health information needs (both *P* < 0.01). Patients' likelihood of reporting excellent, very good, or good health via IVR increased during their participation in the program, from a probability of roughly 64% in week 1 to 88% in week 12, see **Figure 3**. When patients reported good, very good, or excellent health during an IVR call, during the same call they also were more likely to report that they took their medication exactly as prescribed (*P* < 0.001). Specifically, 61.5% of patients reporting good, very good, or excellent health also reported always taking their medication as prescribed in the past week when compared to 45.9% of patients reporting fair or poor health. In contrast, when

**FIGURE 2 |The proportion of patients who reported (left) fair or poor general health and (right) medication adherence problems at least once via IVR, within groups defined by that patient's baseline survey reports (X axis)**. Baseline medication adherence was measured using the Morisky adherence scale, with scores collapsed into groups as follows: "Good" = 0, "Fair" = 1–2, and "Poor" = 3–8.

patients reported good, very good, or excellent health, they were less than half as likely as patients with fair or poor health to report that they rarely or never took their medications as prescribed in the last week (14.4 versus 29.9%; *P* < 0.001).

Despite these encouraging findings, a large number of completed IVR calls generated clinician alerts. Overall, 52.8% of all completed IVR calls generated a clinician alert, with 20.6% of all completed calls generating an alert for fair or poor medication adherence,and 38.7% of assessments to patients with hypertension generating a clinician alert for high or low systolic blood pressure levels. Fair or poor medication adherence was reported 37.2% of the time among patients with diabetes plus hypertension, 20.3% of the time by patients with hypertension alone, and 15.9% of the time among patients with diabetes but not hypertension.

## **QUALITATIVE FOLLOW-UP INTERVIEWS**

Of the 20 IVR program participants who completed follow-up interviews, nearly all (19/20 or 95%) rated the program as either excellent or very good, and most patients reported that the program satisfied the majority of their needs (84%). Almost all patients reported that they would recommend the program to a friend (19/20 or 95%), and 14 of 20 patients (70%) were very satisfied with the amount of assistance they received via the IVR calls. If the program were offered in the future, 84% (17/20) reported that they would definitely use it again. In open-ended questions, most patients reported that they had learned something from the program, either about measuring their blood pressure, remembering to take their medication, or improving their diet. Several patients (8/20 or 40%) reported being pleased that they were reminded to take their medications. In addition, nine patients (45%) noted that they liked the recommendations for improving their self-care.

When asked about weaknesses of the program, most respondents reported that they could not think of any. However, six patients noted that sometimes the telephone connection made it difficult to hear the recorded messages. Three patients had difficulty in responding to the questions, either because they could not keep up with the speed of the call or because their mobile phone was malfunctioning. Five patients reported that they wished that they could have spoken directly to a healthcare provider. Four patients would have liked to be able to change the days and times on which they received the calls without having to contact the project team, and four patients would have liked more information on topics such as diet and other diseases.

## **DISCUSSION**

## **SUMMARY OF FINDINGS**

Here, we describe our efforts to transfer an m-health platform from a research center in the US to an LMIC under the direction of that country's health ministry and in collaboration with academic partners in both countries. The project was the first of its kind in Bolivia. Although Bolivia faces significant infrastructure challenges, we found that the establishment of an m-health platform was technically feasible. Most adult patients with diabetes or hypertension recruited in primary care clinics and health fairs were frequent mobile phone users, including patients who had advanced ages, significant chronic disease burden, limited educational attainment, and indigenous ethnicity. We found that patients were willing to enroll in a program of m-health self-care support and were able to successfully engage with the program, providing potentially valuable health and self-management information to their healthcare team. Patients' probability of reporting good, very good, or excellent health via IVR increased substantially over the 12 weeks of their participation. While we cannot assert that these improvements were causally related to program participation, we did find that when patients had more positive health reports, they also were more likely to report that they were

taking their medication exactly as prescribed. Medication adherence was a primary focus of the educational messages in the IVR calls.

More generally, this project represents a successful case study for how international research projects focused on m-health can be collaboratively developed with low-resource countries. Despite unreliable Internet access in Bolivia, we were able to establish a trans-national team of investigators and computer engineers, including m-health researchers at the University of Michigan, two units within a major university in La Paz (UCB), the regional governmental lead for NCD policy-making and service delivery (SEDES), and four hospitals from which NCD patients were recruited. Through this and prior projects conducted in Honduras and Mexico, we have identified a variety of strategies for transferring m-health technology internationally that may help these initiatives become more scalable and robust elements of NCD management programs in LMICs.

## **LESSONS LEARNED**

In this project, IVR calls were only available in Spanish. However, 37% of survey respondents with diabetes or hypertension reported speaking a non-Spanish language in the home, and an estimated 35% of people in Bolivia speak the indigenous languages of Aymara or Quechua (45). Without translating programs into the language with which patients are most comfortable, we may be missing opportunities to have a meaningful impact on their NCD self-care. Study participants identifying as Aymara or Quenchua speakers were all able to complete the survey in Spanish. However, this language difference may have introduced some inaccuracies in their responses. Moreover, we found that some indigenous women were skeptical about the study and greater inclusion of surveyors from these communities could be a useful strategy to increase their representation and engagement.

We found that during the enrollment process, older adults tended to have greater difficulties understanding how to use the IVR system and blood pressure home monitoring device. Family members often accompanied these patients and assisted them in asking questions. Inclusion of younger family members in the recruitment process was helpful, although as indicated by **Figure 1**, barriers still existed in older adults' program engagement. Greater involvement of family caregivers may be useful for increasing program participation, completion of frequent m-health contacts, and the accuracy of patient reported health and self-care information.

Occasionally, patients reported difficulties completing the IVR calls, because computer server or connectivity problems affected the system in general, or technical issues limited to specific cell phone carriers posed a barrier to call completion. Specific subgroups of patients had unique difficulties, e.g., difficulty with hearing or comprehension among older adults. To understand these barriers and how they could be addressed, we followed up with patients, via phone or in-person meetings. We found that system-related problems were often fixable, in part because the IVR platform was designed to generate detailed reports on the calling process, flagging activity that was out of the ordinary. As novel m-health services are scaled up, a comprehensive and responsive program of technical assistance will be important to encourage enrollment and discourage attrition. For example, it would be beneficial to establish a toll-free help line for patients who have questions or suggestions for improving the service, something that was not done in the current study. Also, completion of a test call upon enrollment may ensure technical functionality and patients' understanding of the system.

Universities such as UCB typically do not have the infrastructure for scaling and maintaining large m-health programs. For programs such as this one to be sustainable in LMICs, it will be important to either engage cell phone providers and/or build technical capacity through government agencies charged with addressing NCD management. In this ongoing program in Bolivia, conflicting financial incentives and differences in organizational cultural between public and private organizations have continued to be challenging in efforts to bring programs such as this one to scale.

Patients generated clinical alerts during more than half of the weeks in which they completed an IVR call. High rates of alerts may reflect pent-up need for care associated with barriers to accessing NCD treatment and self-management support. Nevertheless, alerts generated at this rate likely would make a service such as this one unacceptable to outpatient teams in LMICs who typically have limited resources even for managing in-person visits. It may be that alert rates would decrease over time as patients learn new selfmanagement skills by participating in the m-health program. Also, it may be that some alerts were generated erroneously because of patients'limited levels of educational attainment and lack of familiarity with the IVR calls or home blood pressure monitoring. If the latter is true, more time spent in initial training could pay off yielding lower alert rates. Finally, in an environment in which patients may perceive that they have very limited access to clinicians, some patients may use the m-health service as a "call button," intentionally over reporting health problems in the hopes of speaking directly with their physician for follow-up. Additional research will be important to understand the reasons for high alert rates and how m-health systems (as well as training programs for users) can be designed to keep clinician alerts from over-taxing scarce human resources while improving patients' access to between-visit support.

## **OTHER ISSUES**

As shown in **Figure 2**, 46% of patients who reported good, very good, or excellent health at baseline reported fair or poor health at least once during their IVR follow-up. This highlights two potential benefits of m-health monitoring. First, several studies have shown that patients may under-report health problems (46) as well as potentially stigmatizing behaviors during in-person interviews when compared to automated assessments. Identifying these problems via IVR may serve to more effectively focus clinicians' attention on patients who need assistance to prevent worsening health. Also, chronic diseases often have a waxing and waning course with important changes in patients' symptoms and physiological risk factors occurring in potentially unpredictable ways. For that reason, it is perhaps not surprising that patients reporting very good health when visiting ambulatory care might be in poor health weeks later. The current study suggests that this may be fairly common among patients with diabetes and hypertension in LMICs, and that regular between-visit follow-up via IVR or other

m-health tools could be useful to catch emerging health problems before they become acute.

In future work, we plan to continue to expand the clinical foci of these m-health interventions in Bolivia, including other chronic health conditions that are priorities for LMICs. For example, mental health disorders account for a greater share of the global disease burden than HIV/AIDS, tuberculosis, or diabetes; (47) and depression is the second greatest contributor to disability worldwide (48–52). m-Health programs may facilitate systematic monitoring of patients' depressive symptoms, promptly identifying clinically significant events, and providing tailored psychoeducation (53–56). We currently are working to install new modules as part of the Bolivian IVR platform that address the needs of patients with depression, so that the Bolivian Ministry of Mental Health can evaluate their potential utility for improving mental health service delivery and preventing suicide (53–56). While establishing an independent m-health service in Bolivia is important, it still falls short of the type of international platform required so that m-health knowledge and computing resources can be widely shared across multiple LMICs. We currently are exploring options for establishing that international platform using networks such as the Virtual Campus for Public Health (VCPH, see http://www.campusvirtualsp.org). Twelve Latin American countries participate in the VCPH, and it includes an array of informatics tools for creating and disseminating knowledge bases using Moodle (Modular Object-Oriented Dynamic Learning Environment), an open-source platform used in 235 countries.

## **LIMITATIONS**

We were unable to definitively measure the impact of the m-health service on patients' health, since the study was not a randomized trial and had only a 12-week follow-up. Feedback on participants' satisfaction and perceptions about the program was limited to semi-structured telephone interviews with 20 IVR pilot study participants. As a consequence, we are limited in our ability to evaluate the intervention's impact on subgroups of service users or on important outcomes such as patients' self-care or health service use. These interviews also were not recorded, transcribed, or analyzed using rigorous qualitative methods, and potential positive feedback or concerns may have been missed. However, two research assistants took detailed notes on each interview, limiting the possibility for missed comments during those conversations. Participants with hypertension were given home blood pressure monitors, and while this may be important to consider as a way to merge m-health messaging with impactful self-care tools, home monitors are rarely available currently in LMICs. No single study can establish the feasibility or utility of m-health programs given the variability of LMICs worldwide. Rather, m-health trials will continue to need evidence for their effectiveness in multiple locations that represent the vast diversity of patients, health systems, and capacity for sustaining large m-health services.

## **CONCLUSION**

With these caveats, the present study provides important evidence regarding the feasibility and potential benefit of establishing an independent m-health service in an LMIC. Through active partnerships with leaders in the target country, international collaborators can develop and implement programs for m-health management of chronic diseases. Patients in LMICs who have limited education and diverse socio-ethnic backgrounds can engage successfully with new m-health programs. NCD patients in LMICs can report reliable information during IVR monitoring and selfcare support calls, and there is some evidence that such calls can improve their health status. Researchers and policy-makers should continue to explore options for establishing internationally accessible platforms for disseminating knowledge and technical resources for establishing m-health solutions in LMICs.

## **ACKNOWLEDGMENTS**

In addition to the listed authors, the University of Michigan 2013 Bolivian m-health Summer Internship includes: Evan Hunter Davis and Mirelle Mann (University of Michigan School of Public Health), Zahra Dawson, Elizabeth Feenstra, Gregory Powell, Cody Schultz, and Jonathan Waldmann (University of Michigan School of Medicine) and Aishat Funmi Giwa (University of Michigan School of Information). We thank the following people in Bolivia for their support: Dr. Nenndy Vazquez; Dr. Jose Marcelo Huayta Soto; Dr. Saúl Calderón Campero; and Paola Zapata. John Piette is a VA Senior Career Scientist. Additional support for the study came from: grant number P30DK092926 from the US National Institutes of Health; the UM School of Public Health; the UM International Institute; the UM School of Information; el Servicio Departamental de Salud in La Paz, Bolivia; la Universidad Católica Boliviana; la Universidad Mayor de San Andrés, and la Universidad Pública de El Alto. We also thank the staff and patients of the following hospitals who participated in this study: Hospital La Paz, Hospital Holandés, Hospital de Clínicas, and Hospital San Gabriel.

## **SUPPLEMENTARY MATERIAL**

The Supplementary Material for this article can be found online at http://www.frontiersin.org/Journal/10.3389/fpubh.2014. 00095/abstract

## **REFERENCES**


factors and risk factor clusters in 21 regions, 1990–2010: a systematic analysis for the Global Burden of Disease Study 2010. *Lancet* (2012) **380**:2224–60. doi:10.1016/S0140-6736(12)61766-8


of low-income patients with diabetes. *Diabetes Care* (1999) **22**(8):1302–9. doi:10.2337/diacare.22.8.1302


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 23 April 2014; accepted: 09 July 2014; published online: 13 August 2014. Citation: Piette JD, Valverde H, Marinec N, Jantz R, Kamis K, de la Vega CL, Woolley T and Pinto B (2014) Establishing an independent mobile health program for chronic disease self-management support in Bolivia. Front. Public Health 2:95. doi:10.3389/fpubh.2014.00095*

*This article was submitted to Public Health Education and Promotion, a section of the journal Frontiers in Public Health.*

*Copyright © 2014 Piette, Valverde, Marinec, Jantz, Kamis, de la Vega, Woolley and Pinto. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

## The RAFT telemedicine network: lessons learnt and perspectives from a decade of educational and clinical services in low- and middle-incomes countries

## **Georges Bediang\*, Caroline Perrin, Rafael Ruiz de Castañeda,Yannick Kamga, Alexandre Sawadogo, Cheick Oumar Bagayoko and Antoine Geissbuhler**

Department of Radiology and Medical Informatics, Faculty of Medicine, University of Geneva, Geneva, Switzerland

#### **Edited by:**

Richard Wootton, University Hospital of North Norway, Norway

#### **Reviewed by:**

Debbie Humphries, Yale School of Public Health, USA Charles D. Treser, University of Washington School of Public Health, USA

#### **\*Correspondence:**

Georges Bediang, Department of Radiology and Medical Informatics, Faculty of Medicine, University of Geneva, Geneva, Switzerland e-mail: bediang@yahoo.com

**Background:**The objectives of this paper are to (i) provide an overview of the educational and clinical experiences of the Réseau en Afrique Francophone pour laTélémédecine (RAFT) network, (ii) analyze key challenges and lessons learnt throughout a decade of activity, and (iii) draw a vision and perspectives of its sustainability.

**Methods:**The study was carried out following three main stages: (i) a literature review, (ii) the analysis of key documents, and (iii) discussions with key collaborators of the RAFT.

**Results:** Réseau en Afrique Francophone pour laTélémédecine has been offering an important quantity of educational, clinical, and public health activities during the last decade. The educational activities include the weekly delivery of video-lectures for continuing and post-graduate medical education, the use of virtual patients for training in clinical decision making, research training activities using ICTs and other e-learning activities. The clinical and public health activities include tele-expertise to support health professionals in the management of difficult clinical cases, the implementation of clinical information systems in African hospitals, the deployment of mHealth projects, etc. Since 2010, the RAFT has been extended to the Altiplano in Bolivia and Nepal (in progress).

**Lessons Learnt and Perspectives:**Important lessons have been learnt from the accumulated experiences throughout these years.These lessons concern: social and organization, human resources, technologies and data security, policy and legislation, and economy and financing. Also, given the increase of the activities and the integration of eHealth and telemedicine in the health system of most of the countries, the RAFT network faces many other challenges and perspectives such as learning throughout life, recognition, and valorization of teaching or learning activities, the impact evaluation of interventions, and the scaling up and transferability out of Africa of RAFT activities. Based on the RAFT experience, effective integration and optimum use of eHealth and telemedicine in low- and middle-income countries (LMICs) health systems should take into account the context (resources, infrastructure, and funding), the needs of key stakeholders, and the results derived from theoretical and practical experience.The relevant items highlighted to illustrate the sustainability of the RAFT network and the analyses performed in this study, should serve as discussion basis for the development of eHealth and telemedicine in LMICs.

**Keywords: telemedicine, eHealth, low and middle income countries, Africa, telemedicine network**

## **BACKGROUND**

Health systems in Sub-Saharan low- and middle-income countries (LMICs) face particularly high burden of diseases with poor infrastructure and sanitary equipment, and scarce numbers of trained health professionals (1). These health professionals are mainly concentrated in urban areas and those in rural areas are professionally isolated (2). In addition, geographical and/or financial limitations make quality care widely inaccessible in these countries. Important distances must be covered in some cases to consult a specialist with not negligible risks and costs for the patient, especially considering the quality of the roads and transportation services in Africa.

Telemedicine and eHealth are increasingly used to improve quality and access to healthcare in LMICs (3–7). The Réseau en Afrique Francophone pour la Télémédecine (RAFT) network (8) was created in Africa more than a decade ago to support health professionals working in "medical deserts" (9–11).

This paper overviews the educational and clinical activities of the RAFT analyses lessons learnt throughout a decade of activity and draws perspective on future challenges and sustainability.

## **RAFT NETWORK**

The RAFT (8) is a telemedicine network created by the University hospitals and the University of Geneva (HUG and UNIGE) in French-speaking Africa (Mali) in 2001. In 2008, it extended to English-speaking Africa and is currently being deployed in Portuguese-speaking Africa (Angola). The RAFT is primarily African with 13, 5, and 1, French, English, and Portuguesespeaking countries, respectively. However, it has become more global with the adhesion of Bolivia and Nepal since 2010 and 2014, respectively. The RAFT includes 60 active sites<sup>1</sup> and it connects hundreds of health professionals around the world. Connections are generally established between central and university hospitals, and district hospitals. The management is decentralized with a local team in each country responsible for the strategy, the implementation, and the coordination of activities at the national level (10). A small central team in Geneva ensures the general coordination (10). The RAFT is a collaborating center of World Health Organization (WHO) for eHealth and telemedicine and collaborates with other institutions such as the Université Numérique Francophone Mondiale (UNFM).

## **METHODS**

We used general and specific sources of information to understand the past, present, and future of the RAFT network and identify its difficulties, successes, and failures. First, we reviewed literature (scientific papers mostly) in search for equivalent initiatives to the RAFT in LMICs using PubMed and WHO Databases. Second,we reviewed reports,workshops proceedings,scientific papers and tools (software) to obtain specific quantitative and qualitative information on the RAFT activities (i.e., projects, participation, deployment status, etc.). Third, we held informal discussions with key experienced members (local and international coordinators, national focal points) of the RAFT.

## **ACTIVITIES**

## **EDUCATIONAL ACTIVITIES**

## **Distant continuing medical education for healthcare professionals**

The continuing medical education (CME) targets health professionals in remote areas and is based on live video-lectures of approximately 45 and 15 min of discussions. The sessions address a wide range of medical topics (e.g., general medicine, malaria, HIV– AIDS, diabetes) in French, English, Spanish, and/or Portuguese. The French program is the most complete one with on average 2 h of teaching per week.

## **Post-graduate medical education for residents**

The post-graduate medical education (PME) targets residents in universities and teaching hospitals in French-speaking Africa. Video-lectures (45 and 15 min of discussions) are taught by local academic experts and address topics in the field of surgery, ophthalmology, internal medicine, gynecology, pediatrics, etc. The central goal of PME is to produce and share material of high quality for training specialists in Africa.

## **CME and PME: dudal and participation**

Video-lectures are produced with Dudal, a distant education software developed for the RAFT to operate with low-bandwidth connections (25 kb/s). It allows participation in educational sessions either as listener or lecturer (12).

Seven hundred thirty-two video-lectures have been broadcast through the RAFT during 2007–2013. Seventy-five, 15, and 10% of them are in French, English, and Spanish, respectively. Today, more than 80% of the French and English video-lectures are produced by African experts. Seventy-five and 25% of them are for CME and PME, respectively. In average, 15 and 3 different sites are connected live during CME and PME video-lectures, respectively, gathering hundreds of health professional in one single video-lecture (in the case of CME).

Although live sessions enable a direct "student–teacher" interaction online, an increasing number of participants choose to watch the pre-recorded video-lectures (possibly because they can be watched at any time of convenience). Lectures are completely free and can be downloaded in several formats.

## **Virtual patients for training on clinical decision-making**

Virtual patients are interactive computer simulations of clinical scenarios for medical training (13). Virtual Internet patient simulator (VIPS) is a stand-alone web application initially developed for improving skills of general practitioners in Switzerland, but that was then also used in African countries of the RAFT (14). Studies including African participants, showed that VIPS educational model [paradigm of "blank sheet" (15)] is more suited for learners with some previous experience in the collection of relevant information for decision making (16) and has the potential to improve clinical skills (17).

## **Collaboration with other projects**

Africa Build (AB) is a coordination action aiming to support and develop advanced centers of excellence in healthcare, education, and research in the African countries, through Information Technologies (ITs) (18, 19). Dudal was implemented in the AB learning platform (AB Portal) enabling the production and following of courses globally (AB Portal has 14 e-learning courses, 41 discussion groups, and more than 500 users).

Integrated management of childhood illness computerized training tool (ICATT) is an innovative software to support the implementation of the IMCI (WHO/UNICEF strategy) (20). It was developed by the Novartis Foundation in collaboration with the WHO and the Swiss Institute of Tropical Medicine and Public Health. ICATT targets health workers involved in the management of under-5 years children diseases. A multicentric study is currently being developed in Burkina Faso, Cameroon, and Mali to evaluate the impact of ICATT (2 months of self-training with 2 days of clinical practice at the end of fourth week and 2 at the end of eighth week) on acquired skills by health workers and costs compared to traditional training (11 days workshop with 4 days of clinical practice). To support and stimulate learners in ICATT group, e-mail, or short-message service (SMS) reminders are sent at least once a week and the RAFT is used to deliver weekly video-lectures and to foster interactions between them and teachers.

## **CLINICAL AND PUBLIC HEALTH ACTIVITIES Tele-expertise activities**

Tele-expertise is the second main activity of the RAFT and Bogou is the tool used for it (21). Physicians located in remote and

<sup>1</sup>http://raft.g2hp.net/sites/

infrastructure limited settings can discuss distantly with specialists from developed areas ensuring a better management of complicated cases (e.g., avoiding unnecessary evacuations). Bogou is a web-based application with secured access through a personal login and password. Medical discussions around cases take place within so called circles (community of recognized professionals). All data in Bogou are encrypted using a public and private key system. Bogou can also operate with digital imaging and communications in medicine (DICOM) images for tele-radiology and tele-ultrasonography.

Overall, there are more than 200 members from 18 countries in Africa, South America, and Asia. There are 22 circles that address cases in general medicine, surgery, infectious diseases, radiology, hematology, endocrinology, cardiology, and gynecology. Since 2012, a total of 471 cases have been posted through Bogou. Promising tele-expertise activities are planned for 2014 in surgery in Ghana and pediatrics in Burkina Faso, Senegal, Ghana, Kenya, and Tanzania, in both cases in the context of different collaborations with two international pharmaceutical companies. In many RAFT countries, these tele-expertise activities are coupled with ultrasound or electrocardiography, etc. For instance, Mali, Cameroon, Ivory Coast, and Madagascar have already implemented tele-ultrasonography and/or tele-cardiology. The large-scale deployment of these activities is planned in Mauritania and Angola with the support of the RAFT.

## **Clinical information systems in African hospitals**

There is an increasing demand within the RAFT for the implementation of clinical information systems to improve the collection and management of clinical information in hospitals. Accordingly, two projects based on Open Source systems (22) are in progress at the Mother and Child Hospital in Mali [Cinzan project (22)] and at the Yaoundé Central Hospital in Cameroon [MatLook project (23)].

## **mHealth to support healthcare in Africa**

There is an increasing interest in mHealth in public health activities worldwide. There are two mHealth projects in the RAFT: (i) the use of SMS for fast data collection at health district level in Mali and (ii) the impact evaluation of SMS reminders in tuberculosis (TB) cure rate (Cameroon) (24). The objective of this study is to evaluate the effect of SMS reminders on the cure rate of patients with sputum positive pulmonary tuberculosis (TPM+), measured using 6-month bacilloscopy.

## **RAFT Altiplano**

The Altiplano project (25, 26) aims at replicating the success of the RAFT in Sub-Saharan Africa in isolated regions in the Bolivian Altiplano. This project builds on previous experiences in Africa and the links between the HUG, UNIGE, and Bolivia in medical informatics. The focal point of this project studied at the University of Geneva where his received a Ph.D. thesis in computer science. The RAFT Bolivian network includes 16 active sites and further extensions will take place in 2014. The tools of the RAFT were translated into Spanish and were setup by the Bolivian technical team.

## **LESSONS LEARNT**

Despite the immaturity of activities in some countries (27), the experiences accumulated throughout these years provide both general and specific lessons for each national or local context (28) to promote sustainable implementation of eHealth and telemedicine.

## **SOCIAL AND ORGANIZATION**

The development of the RAFT in each country relies primarily on a network of motivated individuals with an interest in eHealth and telemedicine. The RAFT has become an effective multilateral support network in the area of eHealth and telemedicine favoring South–South collaborations for continuous education and management of patients, and the implementation of eHealth activities and telemedicine. Annual coordination workshops with participants from the entire RAFT network are organized in Africa to support the spirit of sharing, strengthen inter-personal links, evaluate the development of the network, and make decisions about future activities (29). Both the social engineering and decentralized management model of the RAFT optimize the management activities (29).

## **HUMAN RESOURCES**

The RAFT builds capacity among health professionals in Africa and supports their activities with different tools (7, 11). In addition, mobility within the RAFT network including the participation in full training programs in eHealth and medical informatics at the UNIGE, has been crucial not only for the participants themselves but also for their colleagues in their home countries. Trained participants act as knowledge multipliers once back in their countries by building competent local teams. This is the case of Mali and Centre d'Expertise et de Recherche en Telemedecine et E-Sante (CERTES). Staffed with a dozen healthcare and IT professionals, the CERTES trains healthcare professionals to use health IT tools, provide operational support for telemedicine activities and health information systems deployment, and run research projects financed by competitive funds (30). This same strategy is also being adopted in Cameroun. The goal is that these centers operate at the subregional level supporting neighbor countries in eHealth and telemedicine activities.

## **SOFTWARE DEVELOPMENTS AND DATA SECURITY**

The RAFT tools (Dudal and Bogou) have been developed with Java to run on several operating systems and work reliably with unstable and low-bandwidth connections (e.g., Java web start technology, compression of documents and images, use of text messages for asking questions, data storage under stable and secured environments).

Video-lectures produced with Dudal can be converted to HTML5 making possible to follow them regardless of the web browser. They can also be watched offline after downloading.

Patients data in Bogou are completely secured. Users must go through a validation of their registration demand and then access the system with a username and password. Users must be accepted within a specific circle to be active on the system. Medical data are encrypted via the system of asymmetric encryption public key – private key.

## **POLICY AND LEGISLATION**

One of the major challenges of the RAFT is to align eHealth and telemedicine activities with local needs and national health strategies. Besides the implication of local actors in the conception and implementation of the activities, the RAFT actively establishes new partnerships with national (e.g., Ministry of Health) and international institutions (e.g., WHO) and/or Non-Governmental Organizations (e.g., UNFM,Agence Universitaire de la Francophonie). This is an anchoring approach both institutional (top-down) and end-user appropriation (bottom-up). The RAFT aims at guiding and supporting countries to progressively implement national strategies, policies, and regulations for eHealth and telemedicine.

Some countries are now developing national eHealth strategies and RAFT local teams are usually involved in this process. In Burkina Faso, the RAFT medical coordinator is now in charge of eHealth at the Ministry of Health. In Ivory Coast, the RAFT focal point has been mandated by the government to lead the development of an eHealth strategy. In Congo-Brazzaville, Mauritania, and Niger, the RAFT model is used as a pilot for the implementation of the national program.

The RAFT also favors the emergence of scientific associations and societies (31, 32) in the context of academia or others sectors that can contribute to the development of eHealth and telemedicine activities and to their transcendence into the national political field. These scientific societies generally led national, regional, or international congresses and workshops gathering the key national scientific and political actors. Mali, Cameroon, Ivory Coast, Tchad, Madagascar among others offer good examples of such societies.

## **ECONOMY AND FINANCING**

## **Financing of the implementation of the RAFT sites and sustainability**

After more than a decade of activities, the financial sustainability of the RAFT still remains a major issue (33). Long-term funding is necessary to ensure the involvement of care professionals both as users and providers of quality contents. To motivate a sustainable financial development, the RAFT agrees to provide 2-years worth of connection fees, assuming that the deployment contact will require the receiving hospital to guarantee the financing of these connection costs after the two first years of operation (29). These 2 years are usually sufficient to document the benefits of the activities, and convince decision makers to cover the operational costs. The final goal is to incorporate these activities in governmental plans.

## **Tele-expertise business model**

To achieve financial independence of the hospitals and ensure sustainability, a business model is deployed for the tele-expertise service in some of the network nodes, which creates a win–win situation for local physicians as well as the patient. The patient pays for the service locally, thus, avoiding to pay additional transportation costs if it should be done to the capital. Part of this payment is used to finance devices used in the activities or the Internet connection, while the rest is used to pay the treating physician as well as the physician in the reference hospital (29).

Despite the development of these type of business models and their implementation in certain pilot sites, their scaling up is still limited within the RAFT. Numerous institutions are unable to sustainably ensure independent funding for their sites. Business models may be particularly difficult to implement in contexts with low activities. These could be related with the novelty of these types of activities and the immaturity of the telemedicine market in developing countries (34). In these settings, there is still a digital divide and telemedicine targets mainly individuals and not organizations or institutions as in developed countries, which does not allow the economy of scale and higher profits (34). A possible immediate solution could be the creation of national clusters that could drain a critical mass of patients to justify and support the use of resources (equipment, Internet connections, experts' time, etc.).

However, for effective development of eHealth and telemedicine, it should be put in place a more comprehensive business plan that integrates the context, the needs of stakeholders involved, the technologies-based interventions with high-added value for the improvement of practice and finally, the funding model (35).

## **PERSPECTIVES**

eHealth and telemedicine can potentially improve health systems in LMICs (4, 36–39) and therefore the RAFT has gained interest over the years. The RAFT network counts today with a solid experience in LMICs to keep building on to improve access to quality care.

## **LIFELONG LEARNING**

The RAFT offers health professionals a range of tools and resources (e.g., e-learning material, tele-expertise platform, simulation tools) to ensure their learning through all stages of their career and life. This potentially enhances professional motivation, cohesion, and sustainability of the RAFT network.

## **RECOGNITION AND VALORIZATION**

Recognition and valorization of teaching and learning is of major interest in LMICs. Accordingly, Conseil Africain et Malgache pour l'Enseignement Supérieur (CAMES) formally recognizes the production of RAFT courses as an academic merit for career promotion. A new RAFT platform has been recently developed to make the learning experience more complete including learning material to complement video-lectures (e.g., articles, videos) and MCQs. Certificates of participation to the courses can be provided.

However, the main challenge remains to integrate these contents in formal programs of continued education or of postgraduate studies with a full recognition in form of ECTS credits.

## **IMPACT EVALUATION**

Impact evaluation of the RAFT activities remains an "unfinished business"(40–43),and therefore,it must be the priority for the next years. We aim at measuring the impact of the RAFT on clinical (patient), public health (population, health system), and socioeconomic (patient, health system) domains, in terms of processes and outcomes [e.g., Mali (44), Cameroon]. This goal is also the major demand by decision makers and donors (45–49). To achieve this one, it is required to develop good methodologies for optimal impact assessment of eHealth and telemedicine activities before.

## **SCALING UP AND TRANSFERABILITY OUT OF AFRICA**

Although the RAFT is active in a considerable number of countries of the world, the number of sites per country is low and they are mainly concentrated in reference hospitals in urban areas. A larger number of sites and a wider distribution in the periphery are needed to improve health systems.

Due to infrastructure and economic reasons, the best level of deployment is the district hospital. District hospitals generally cover an important population (50,000–200,000 inhabitants), include a critical mass of health professionals (2 or more physicians) and have medico-technical infrastructure such as laboratories and operating room and associated activities that cover primary cares. 3G Internet connections can be easily obtained for reasonable costs.

The RAFT is built on basic and general principles and ways of operation that respond to important and common needs around the globe, making it potentially transferable. The success of the RAFT in the Bolivian Altiplano and the promising progresses made in Nepal during the last months evidence that RAFT is not only African but also potentially global. However, possible local needs and socio-cultural and political differences even within the same geographical sub-region must be carefully considered and adapted to them.

## **REFERENCES**


49. Scott R.E, Saeed A. *Global eHealth-Measuring Outcomes: Why, What and How. A Report Commissioned by the WHO Global Observatory for eHealth*. Bellagio (2008). Available from: http://www.ehealth-connection.org/files/ conf-materials/Global%20eHealth%20-%20Measuring%20Outcomes\_0.pdf

**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 21 May 2014; accepted: 21 September 2014; published online: 07 October 2014.*

*Citation: Bediang G, Perrin C, Ruiz de Castañeda R, Kamga Y, Sawadogo A, Bagayoko CO and Geissbuhler A (2014) The RAFT telemedicine network: lessons learnt and perspectives from a decade of educational and clinical services in low- and middle-incomes countries. Front. Public Health 2:180. doi: 10.3389/fpubh.2014.00180*

*This article was submitted to Public Health Education and Promotion, a section of the journal Frontiers in Public Health.*

*Copyright © 2014 Bediang , Perrin, Ruiz de Castañeda, Kamga, Sawadogo, Bagayoko and Geissbuhler. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

## Tele-education in South Africa

## **Maurice Mars\***

Department of TeleHealth, Nelson R Mandela School of Medicine, University of KwaZulu-Natal, Durban, South Africa

#### **Edited by:**

Richard Wootton, University Hospital of North Norway, Norway

#### **Reviewed by:**

Daniel Francis Fahey, California State University San Bernardino, USA Colin Stephen Chandler, University of Edinburgh, UK

#### **\*Correspondence:**

Maurice Mars, Department of TeleHealth, Nelson R Mandela School of Medicine, University of KwaZulu-Natal, Pvt Bag 7, Congella, Durban 4013, South Africa e-mail: mars@ukzn.ac.za

**Introduction:** Telemedicine includes the use of information and communication technology for education in the health sector, tele-education. Sub-Saharan Africa has an extreme shortage of health professionals and as a result, doctors to teach doctors and students. Tele-education has the potential to provide access to education both formal and continuing medical education. While the uptake of telemedicine in Africa is low, there are a number of successful and sustained tele-education programs. The aims of this study were (i) to review the literature on tele-education in South Africa, (ii) describe tele-education activities at the University of KwaZulu-Natal (UKZ-N) in South Africa, and (iii) review the development of these programs with respect to current thinking on eHealth project implementation.

**Method:** A literature review of tele-education in South Africa was undertaken. The development of the tele-education services at UKZ-N from 2001 to present is described. The approaches taken are compared with current teaching on eHealth implementation and a retrospective design-reality gap analysis is made.

**Results:** Tele-education has been in use in South Africa since the 1970s. Several forms of tele-education are in place at the medical schools and in some Provincial Departments of Health (DOH). Despite initial attempts by the National DOH, there are no national initiatives in tele-education. At UKZ-N, a tele-education service has been running since 2001 and appears to be sustainable and reaching maturity, with over 1,400 h of videoconferenced education offered per year.The service has expanded to offer videoconferenced education into Africa using different ways of delivering tele-education.

**Conclusion:** Tele-education has been used in different forms for many years in the health sector in South Africa. There is little hard evidence of its educational merit or economic worth.What it apparent is that it improves access to education and training in resource constrained settings. The development of local and international tele-education at the UKZ-N has not followed what is currently considered to be best practice but shows how programs can develop if there is a real need and the solution assists in meeting the need. Further work is required to analyze the economics of these tele-education endeavors.

**Keywords: tele-education, videoconference, South Africa, eHealth implementation, telemedicine, review**

## **INTRODUCTION**

The uptake of telemedicine in Africa has been slow. There are many reasons for this – the burden of disease, shortage of doctors, poverty, limited funding of health services, lack of infrastructure, poor but costly connectivity, irregular power supply, lack of political will, and limited computer literacy (1). Telemedicine also adds extra steps to the routine workflow, adding burden to already overworked health professionals. But telemedicine is not confined only to clinical services. The World Health Organization's definition of telemedicine includes the use of information and communication technologies (ICTs) "for the continuing education of health-care providers." (2). This meets the definition of tele-education, which Curran defined "as the application of ICTs in the delivery of distance learning" and noted that the way in which it is delivered is dependent on the technologies and media used (3).

Education is often used as a catch all word to encompass three distinct activities; formal education leading to an academic qualification, training, and acquisition of skills, and raising awareness. In this paper, tele-education will include education in its broadest sense.

One of the perceived benefits of telemedicine is overcoming the sense of isolation experienced by doctors in rural communities. This can be achieved by providing professional support through second opinion consultation, referral, access to specialists and specialist services, and continuing medical education. The benefit of tele-education to rural doctors has been described in Australia and North America (4, 5). The experience in Africa is similar, not only for doctors in rural areas but also doctors in urban areas where specialists may be in short supply. There are several examples of successful and sustained educational activities.

The Reseau en Afrique Francophone pour la Telemedicine (RAFT) program initiated by the University of Geneva in 2001 currently links up to 18 countries in Francophone Africa using low bandwidth videoconferencing (VC). It provides 16 h of education a month with most of the teaching emanating from African countries (6). The Institute of Tropical Medicine in Antwerp uses email, the Web, and mobile phones to provide HIV education (7), and Medical Missions to Children provides daily seminars broadcast three times a day via satellite and the Internet (8). The Pan-African eNetwork between India and many African countries (9), African Medical Research Foundation (AMREF), which is reskilling nurses in central and East Africa (10), and Project Hope providing HIV training between the US and five African countries are other examples of sustained educational activities in Africa (11). Unlike the developed world, much of this education takes place in urban rather than rural areas because of the limited telecommunication infrastructure in rural Africa.

The WHO definition of telemedicine refers to continuing medical education, but what of its role in graduate and undergraduate education?

The extreme shortage of doctors in sub-Saharan Africa is well documented with 26 doctors per 100,000 people against a global average of 141 per 100,000 people and 331 per 100,000 people in the European Region of the WHO (12). Three approaches have been proposed to address the shortage; increase the number of medical graduates, make migration of doctors from the developing world more difficult, and train medical assistants to perform certain tasks.

The number of medical school in sub-Saharan Africa has increased rapidly over the past 15 years. In 2000, the World Directory of Medical Schools recorded 77 medical schools in sub-Saharan Africa (13). By 2006, there were 86 medical schools (14) and the sub-Saharan Africa Medical School Study of 2011 found 169 medical schools, several of which were not yet graduating students (15). The Foundation for Advancement of International Medical Education and Research, which maintains the International Medical Education Directory lists 122 medical schools that are active and recognized by the appropriate Ministry of the country (16). With the extreme shortage of doctors, who is educating and training students in these new and developing medical schools, especially in the specialties?

The situation is no different in South Africa. While South Africa has 77 doctors per 100,000 people this includes, retired and nonpracticing doctors and those who have remained on the register but practice in other countries (12) and it is estimated that the actual figure is nearer 55 doctors per 100,000 people (17). There are 39,800 medical practitioners registered with the Health Professions Council of South Africa (HPCSA) and an additional 14,000 doctors are required to merely fill the vacant posts in the Government sector (18). The eight South African medical schools produce around 1,300 doctors per annum. A new medical school is planned but none have been built since the 1970s when the population of the country was less than half its present number. Doubling the output of the eight medical schools in South Africa and opening a new medical school will merely maintain the current doctor to patient ratio (19). To overcome this, medical schools are increasing their intake but are constrained by physical, human, and financial resources and the number of "teaching beds" in academic hospitals. The government has also been sending students from rural areas to Cuba for medical training and plans to increase the number sent and graduating to 1,000 over the next 5 years (20). A new category of practitioner, clinical associate, is being trained at four of the medical schools to work in district hospitals under supervision of a physician. This is a 3-year bachelor degree program.

In the African and South African context, the WHO's definition of telemedicine needs to be expanded to include undergraduate professional development as it improves access to scarce academic resources.

The aims of this study were to (i) review the literature on teleeducation in South Africa, (ii) describe tele-education activities at the University of KwaZulu-Natal (UKZ-N) in South Africa, and (iii) review the development of these programs with respect to current thinking on eHealth project implementation.

## **MATERIALS AND METHODS LITERATURE REVIEW**

A literature review was undertaken of telemedicine in South Africa. Searches were conducted of Pubmed, Scopus, Cinahl,African Journals online, and African Index Medicus using search terms such as (("South Africa") AND ("tele-education" OR "distance education" OR "videoconference" OR "remote education")). As teleeducation falls under the definition of telemedicine the databases were also searched using common keywords relating to telemedicine. For Pubmed, the terms (("telemedicine" OR "ehealth" OR "mhealth" OR "email" OR "electronic mail" OR "telephone" OR "mobile phone" OR "cell phone" OR "videoconference") AND ("South Africa") AND ("tele-education" OR "distance education" OR "remote education")) were used to search the title, abstract, and keywords of articles indexed. Google Scholar and Google were searched for each of the eight medical schools and nine Provincial Department of Health (DOH) for VC or tele-education.

The abstracts of all papers returned by the searches were read. The full papers of those that appeared relevant were obtained and read. Those reporting the use of ICTs for education in the health sector in South Africa were included in the review. The aim of the review was to document evidence of tele-education in South Africa and no attempt was made to assess the quality of the papers.

## **CASE STUDY**

A single case study was undertaken of tele-education at the UKZ-N. The objective of the study was to describe the evolution of the service and compare its development with current thinking about eHealth implementation. The records, reports, and minutes of meetings related of the service were reviewed.

#### **DESIGN-REALITY GAP ASSESSMENT**

A retrospective assessment of the design-reality gap at the time of implementation of videoconferenced tele-education at the UKZ-N was undertaken using Heeks' Design-Reality Gap Assessment model (21).

## **RESULTS AND DISCUSSION**

#### **LITERATURE REVIEW**

Eighty-three papers and Web pages relating to tele-education in South Africa were found, of which 23 were relevant and summarized. Of these, 16 reported tele-education activities, 5 provided data on tele-education, and 2 referred to the educational benefits of participation in a clinical telemedicine service.

Tele-education in South Africa has followed the evolution of technology and has taken many forms. It can be broadly categorized in terms of continuing medical education, postgraduate education, undergraduate education, and education through participation in telemedicine services.

In the 1970s the South African College of Medicine provided continuing medical education using tape recordings of lectures (22). As South Africa was a late adopter of television in 1975, there was little use of closed circuit television for education, as pioneered in Nebraska for mental health in the 1950s (23). Wessels noted the educational value to the general populations of a television series that focused mental health in 1999 (24). The use of tele-education in nursing and medical courses is not new in South Africa with nursing courses being offered by satellite as early as 1990.

The South African government saw the potential of telemedicine and implemented a National Telemedicine System in 1999. It was to have consisted of three phases. The first, pilot phase, involved establishing telemedicine services at 29 sites in six of the nine provinces. The second phase envisaged connecting the eight medical schools by VC for the provision of both postgraduate and continuing medical education and extending the original project to 76 VC sites. Although there were promising early reports of the system, only a teleradiology store and forward service in neurosurgery was sustained, with most services failing within 2 years (25, 26). As a result, phases two and three were never implemented. In the absence of a Government backed telemedicine system, various Provincial DOHs, Universities, and the Medical Research Council of South Africa continued to support and develop telemedicine and tele-education.

Part of the problem appears to have been that the National Telemedicine System was conceived, planned, and budgeted centrally in the National DOH, with the National Health Information System of South Africa (NHISSA) Committee having oversight. After the installation of equipment in phase one, the Provincial DOHs were expected to take ownership of "their" programs and submit monthly reports to NHISSA. Telemedicine was an additional function within the Health Information units in the Provinces with no budget or posts for telemedicine in place (27).

In Free State Province, the Provincial DOH has used an interactive-learning and communication and management network (ICAM) since 2002 with initial setup costs of ZAR11 million (US\$ 2 million at that time) and a further ZAR12 million spent to upgrade facilities in 2011. The Department currently runs a broadcasting facility including three control rooms, three broadcasting studios, two editing rooms with voiceover booths, and editing equipment. Educational content, lectures, seminars, and practical training sessions presented by DOH staff and academics from the University of the Free State Medical School are filmed, recorded, edited, and broadcast by satellite link to 40 sites in the Province. Two way audio communication is provided through a response keypad. There are approximately 16,000 participants in these sessions annually (28).

At the University of Stellenbosch interactive television was used in the early 2000s to see and hear lecturers, with students interacting by telephone from the receiving studio (29, 30). The Ukwanda Rural Clinical School has a telemedicine and eHealth component with a VC link between the medical school and two rural health-care hostels allowing undergraduate students to receive education while on rotation in rural areas (31). Several postgraduate programs in nutrition and reproductive biology and 11 in nursing are offered at a distance by VC with the University having access to 19 VC venues across the country (32).

In 2006, a nationwide radiology education program for trainees was trialed, initially between three medical schools and later with seven participating via a commercial VC bridge service. Problems were noted with the quality of the radiographic images shared over the network as connectivity defaulted to that of the site with the lowest bandwidth, 128 kbps. To overcome this, images of radiographs were saved to PowerPoint and distributed by email before the scheduled sessions and projected locally with VC used for discussion between the sites (33). A plan to establish shared postgraduate radiology teaching nationally never materialized.

In the Eastern Cape, telemedicine services were initiated in the late 1990s and tele-education by VC was reported in 2001 (34). Banach reported education as part of the Eastern Cape telemedicine activity in 2008 with four health resource centers linked for teaching and case presentation (35).

At the University of Cape Town, the Department of Pediatric Surgery has installed video cameras in their operating theaters so that surgical procedures can be broadcast to lecture theaters (36). They have recently begun sharing weekly seminars with some medical schools in Africa using Internet protocol (IP) based desktop VC. The Department of Gastro-enterology has participated with the Universities of the Witwaterstrand and KwaZulu-Natal in receiving high definition video of endoscopic surgery undertaken and streamed from Japan (37). In 2011, a working group was formed to investigate the use of VC in education at the University and a VC link has been established with a rural hospital.

The University of South Africa is a distance learning university with 19 VC venues in South Africa. No information is available on tele-education in the health sector which includes nursing and public health.

The Mindset Network is a non-profit organization in partnership with the National Departments of Education, Health, Communication, Science and Technology, and communication companies. Started in 2002, it provides educational material to school children – Mindset Learn, teachers – Mindset Teach and the health sector – Mindset Health through digital satellite television channels. Mindset Health provides recorded programing of health related issues for patients and the public which is played in waiting areas of clinics and public hospitals. These are provided via subscription free digital satellite television through Sentech and Intelsat, with equipment installations funded by donors.

The health content covers topics such as prevention of mother to child transmission of HIV and infection prevention and control delivered in the form of dramas, discussions, interviews, documentaries, and magazine shows with content aligned to the National DOH policies and guidelines. The Mindset outreach program provides training to health workers on facilitating use of the Mindset Health video content for formal and informal training in clinics and other community settings. Currently, the Mindset Health Channel is available in 625 public health facilities in the country with a further 788 sites planned as part of the implementation of the National Health Insurance in South Africa. Material is also available on YouTube but use of this facility has been very limited.

Mindset provides an online certificate course, on the Fundamental Management of HIV and Aids for health-care providers, using a learning management system. The course is integrated into the curriculum of nursing schools and is used for continuing professional development. No data are available evaluating use of Mindset Health (38).

After the failure of phase one of the National Telemedicine System in KwaZulu-Natal, the University of Natal used the VC equipment for tele-education and videoconferenced education in radiology began in 2001 with point to point VC at 128 kbps (27). By 2010, this had grown to 35 academic programs broadcasting a total of 123 h of interactive teaching per month with 37 different sites participating in the various programs (39).

In pediatric surgery, weekly seminars have been shared at various times with an academic hospital in the Eastern Cape, a regional hospital in Limpopo Province, neither of which had a pediatric surgeon and recently an academic hospital in Zimbabwe. In the first 4 years of the service the seminar, attended in Durban by an average 13 people, was shared with 63 others at the various sites (40). Surgeons from several other African countries have requested to be included in the videoconference program. In the absence of VC infrastructure at other medical schools in Africa, the VC seminars in Durban have been recorded to digital video disk (DVD) and mailed to four medical schools in Central and East Africa. They have been incorporated into the postgraduate training programs of surgeons and pediatric surgeons and also used in undergraduate medical training. An additional 140 people have access to the seminars through this program (41).

Chipps et al. conducted a systematic review of the effectiveness VC-based tele-education for doctors and nurses and reported equivalence with face to face teaching and an increase in knowledge and knowledge retention reported in one study (42). Videoconferenced teaching for psychiatry registrars was then implemented with 6.5 h of VC education conducted weekly using ISDN connections at 128 kpbs. Over the period of review, there was a significant increase in the number of trainees who elected to join by VC and not travel to the Medical School with concomitant savings in time and transport costs. Audio quality, while satisfactory during presentations was a problem for some during discussions. Participants were satisfied with the quality of PowerPoint presentations but concerns were raised about some teaching aids, notably problems with the colors in drawings. Both local and distant attendees were less satisfied with the use and visualization of specimens in anatomy sessions (43).

To extend mental health education outreach, a set of lectures was videoconferenced to doctors at designated Mental Health Hospitals in KZ-N at 128 kbps. As it was difficult to co-ordinate activities at the different hospitals, the lectures were subsequently recorded to DVD and sent to the hospitals for local use. Pre and post testing of participants' knowledge showed improvements in post test scores (44). A pilot project assessed the feasibility of teaching nurses taking a decentralized educational program by VC and this was later adopted (45).

The American Medical Informatics Association developed the concept of HIBBS, health informatics building blocks, in their approach to increasing the number of informaticians. The HIBBs program aims to create a variety of training modules on health informatics that may be used as open educational resources. Under the direction of the Global Health Informatics Partnership, four HIBBS were developed at UKZ-N, assessed by senior academic informaticians in the US, and made available on the Internet through Open Educational Resources Africa (OER Africa). Each module consists of a PowerPoint presentation, an audio file, a video of the presentation with audio, and a Word document with the full text of the presentation. Pre and post testing with repeat post testing after 6 weeks showed significant improvement in both post test scores *p* < 0.001. The University of Cape Town, a founding partner of OER Africa has made a large number of educational videos and presentations available on the site.

The educational benefits to doctors and nurses participating in clinical telemedicine service have been documented. An early audit of a synchronous teledermatology service in KZ-N evaluated the educational benefit of being able to discuss cases with a dermatologist with the educational value of 86% of teleconsultations rated as good or very good (46). In a store and forward tele-ophthalmology service, it was noted over time that only more complex cases were referred with the referring doctor having learned from the responses obtained from the ophthalmologists for previous cases (27).

Colven quantified this in a study of a store and forward teledermatology service between five primary care physicians and one dermatology trained nurse at five rural sites and the Department of Dermatology at the University of Cape Town. In addition to the diagnosis and management plan, referrers were provided with relevant references. Correlation of diagnostic concordance between the referrer and the dermatologist and the number of referrals over time was significant and increased from 13% for the first four referrals to 50% for the 9th to 12th referrals. Partial concordance, the dermatologists diagnosis included in the referrers' differential diagnosis, also increased significantly over time from 33 to 60% (47). The number of referrals per month decreased significantly over time, and it is not clear whether this was due to improved diagnostic acumen resulting from the service and references provided (48). It was noted that although a clinical referral template was available this was used only 27% of the time, and mostly by the only nurse referrer with doctors including the history in the body of the email sent. Over the 29 months of the service, 4.1 cases were sent per month, which equates to less than one case per site month. This is in keeping with the finding that 61% of telemedicine services world-wide refer fewer than one case per site per week (49).

## **mHealth**

The GSMA website lists 96 mHealth projects in South Africa but none appear to be offering clinical services other than call center advice. There are many examples of SMS services for patient education, appointment reminders, treatment adherence, and patient support.

Woods et al. (50) describe the use of short message service (SMS) text messages for providing ongoing education to midwives. The service was linked to a Website from which additional reading material could be downloaded. The service was well received but only 17 (34%) of 50 interviewees accessed the Website, while 16 (32%) indicated that they wished to access the Website but did not have Internet access (50). This highlights the problems faced in delivering IP based educational material to rural areas. While an assumption is made that smart phone technology will address the problem, the reality is that cellular phone service providers are reluctant to deploy expensive infrastructure for poor people in rural areas who are not able to afford the service.

## **Regulatory issues**

The HPCSA is a statutory body tasked with licensing health professionals, providing guidelines for ethical practice and "to serve and protect the public in matters involving the rendering of health services by persons practicing a health profession." The HPCSA has been working on General Ethical Guidelines for Good Practice in Telemedicine for over 7 years. The most recently available draft version defines telemedicine incorrectly as, "*the exchange of information on health care at a distance for the purpose of facilitating, improving and enhancing, clinical, educational and scientific health care and research, particularly to the under-serviced areas in the Republic of South Africa*" (51). This definition fails to include the use of information and communication technology. As the guidelines propose that written informed consent be required for all aspects of a telemedicine encounter with copies kept by both the provider and recipient, this would include tele-education and will add unnecessary administrative load to those providing tele-education (52).

#### **TELE-EDUCATION IN KWAZULU-NATAL – A CASE STUDY**

Videoconferenced tele-education began in KwaZulu-Natal in 2001. The evolution of this service will be described and then compared with current thinking about eHealth implementation and project risk.

The proposed National Telemedicine System planned to include tele-education in its second phase with the expectation that the eight medical schools would offer shared postgraduate teaching and continuing medical education. An education subcommittee of the National Telemedicine System Committee was formed and tasked with facilitating tele-education. As the second phase was never funded, the medical schools did not receive VC units, nothing came of the plan and the committee was disbanded.

As a member of the sub-committee, the author saw the potential of tele-education to address, in part, some issues facing postgraduate education at the medical school in the University of Natal. The medical school is situated in Durban and has satellite teaching hospitals in the Durban region and several towns up to 300 km away. These hospitals are accredited to provide trainee specialists, registrars/residents, with a portion of their supervised clinical training time. The postgraduate academic programs of all specialties at the medical school include weekly seminars, in many instances weekly journal clubs and research meetings and in radiology, daily teaching sessions, all of which are conducted at the medical school. Staff at distant hospitals find it difficult to attend because of the effect of traveling and participation time on clinical services. Videoconferenced tele-education was seen as a potential solution. In effect, the solution to the problem was proposed without a formal needs assessment.

In KwaZulu-Natal, nine VC sites were established for teleultrasonography and tele-ophthalmology in Phase One of the National Telemedicine System. Site visits were undertaken to determine whether staff at these sites envisaged developing telemedicine services using VC. No services were planned and at several sites, the units had been disconnected and stored. Permission was sought from the Provincial Member of the Executive Committee for Health for the medical school to relocate some of the VC units and use them for post graduate tele-education.

A decision was made to begin on a small scale, learn from the experience, refine the approach, and then scale up services. The heads of two clinical departments, surgery, and radiology were approached, asked to relocate their teaching venues to VC venues and broadcast their teaching sessions to another hospital.

At the outset, a simple principle for the provision of educational material for tele-education was adopted: no-one must do extra work. In the absence of software to transmit a PowerPoint presentation from a computer to the VC unit, the camera in the seminar venue was aimed at the screen on which the presentation was being projected. As technology improved, it was possible to send the presentations directly through the VC unit.

All training support was provided by local technicians. For the first 4–6 weeks of linking to a new site, a technician traveled to the site and ensured that the connections were made and trained local doctors to make connections and perform basic troubleshooting. Support from Durban is also provided by telephone during every VC session. Presenters were encouraged to ensure that participants at distant sites were drawn into discussions and asked questions. PowerPoint presentations posed an unforeseen problem when there were too many lines of text on a slide, fonts were too small and the contrast between text and background was poor. Although guidelines were developed and given to presenters, few complied. The technician attending the session would request the presentation a day in advance and make changes as needed. This was not without its problems.

The number of sites able to participate in a session was limited by the available software and technology, which initially limited it to the send site and one other site. As VC equipment and software improved, it became possible to link to three additional sites and then to five sites. In the absence of a VC bridge, units with multisite capacity were "daisy chained" so that a five port site could use one or more of its ports to link to other three or five port sites, which in turn linked to other sites.

Videoconferenced tele-education grew as per Rogers' diffusion of innovation theory (**Figure 1**).

The number of programs and activities participating in VC teaching has increased and is nearing saturation (**Figure 2**).

In 2005, a postgraduate program in HIV Management was the first academic program designed to be delivered by VC. Lectures were given at the Medical school linked to four regional hospitals in the province. Students went to the nearest site and participated in classes that ran over weekends. Administrative support was provided at each site.

The original VC units installed during the National Telemedicine System were Polycom units with connectivity provided by ISDN lines at 256 kbps. Subsequent installations were at 128 kbps for distant sites and 384 or 512 kbps for multisite units. Use of

**number of videoconferencing bookings per year from 2001 to 2013**.

ISDN was dictated initially by VC equipment and access to bandwidth. As IP based VC units and software became available, ISDN continued to be used because lack of IP bandwidth at hospitals. Until recently, most hospitals in the Province were provided with 128 kbps bandwidth shared between administrative and clinical services.

## **Administration**

Preceding the National Telemedicine System, the then university of Natal and KZ-N DOH formed a joint committee to work on the ICT needs of both sectors at the planned Inkosi Albert Luthuli Central Hospital, a paperless hospital developed under a public private partnership. This served as the precursor to the KZ-N Telemedicine Steering Committee. When the KZ-N component of the National Telemedicine System was devolved to KZ-N DOH, they were expected to administer and submit monthly reports on their telemedicine activity. As the KZ-N DOH did not have a budget for telemedicine and were not organizing telemedicine services the University assumed these functions.

This led to the Joint KZ-N DOH and UKZ-N Telemedicine Steering Committee made up of the IT Manager of the DOH, the three area managers of the Province and three representatives from the Medical School, staff in the Department of TeleHealth. The administrative model was fraught with problems. The KZ-N DOH had not taken ownership of telemedicine or tele-education. Telemedicine was an additional responsibility for the managers concerned and without a budget they had little that they could add of value. While the three area managers understood that the DOH was supportive of telemedicine they had little understanding of what telemedicine entailed in terms of infrastructure, staff and administration. Between 2002 and 2010, awareness sessions were arranged within each of the three areas so that the 11 district managers and the hospital managers would understand why ICT infrastructure had been installed and what telemedicine was planned for their hospitals.

Subsequent studies by Chipps et al. showed limited understanding of both area managers and district managers of what was being planned for telemedicine even though the infrastructure was already in place (53). Few were aware that tele-education was taking place within their jurisdictions.

The Telemedicine Steering Committee met on a quarterly basis for several years and endorsed activities planned and run by the University. Managers, however, lost interest in the committee as it had no budget and there were no posts and people to do the work within the DOH. They also did not see tele-education as part of telemedicine. The situation has recently improved with the appointment of a Telemedicine Manager and four support staff. Their focus remains clinical telemedicine and they do not have a dedicated budget.

#### **Provision of infrastructure**

Additional VC units have been installed over the past 8 years through funding from the Global Fund and a co-operative agreement between the Italian Government and the KZ-N DOH. Each of the 11 districts has at least three units and there are currently over 45 VC units in the province with several at the main teaching hospitals. Each VC unit has a data projector, computer, sound system, and security. Despite security cages, locks and burglar bars, several units, and sound equipment have been stolen. It was agreed that the University would maintain equipment at teaching hospitals and the KZ-N DOH the remainder. No procurement, maintenance or replacement budgets exist, and replacement is dealt with on an *ad hoc* basis.

The need to continue using ISDN for connectivity remains a problem especially in rural areas. The national telecommunications company, Telkom, is the sole provider of ISDN and although the Provincial DOH receives priority maintenance this has not always been forthcoming. On a number of occasions, one line at a site has become faulty and can take several months to be fixed. Services have also been disrupted when hospital administrators do not realize that VC is by phone line and fail to pay the phone line rental bills.

#### **Support**

Onsite technical support for VC is another unresolved problem at the hospitals. Following current best practice of the day in 2000, super users were appointed at each hospital. They ranged from secretarial staff to a store foreman. IT support and VC support was not within their job descriptions and their first priority was to their line manager and allocated work. The Department of Health was not able to provide training on the VC units for the super users. This was done by technicians from the Department of TeleHealth.

The need for dedicated site co-ordinators based on the Canadian model was soon identified. It was suggested that they be nursing staff and that a percentage of their time be allocated to videoconference support for both educational and clinical activities. As nurses work different shift hours and because some teleeducation sessions occur before and after normal work hours and over weekends, it was proposed that three nurses be appointed at each site with up to 25% of their time allocated to telemedicine support. This was agreed to by the IT Steering Committee in 2008 and job descriptions were drawn up. The Provincial DOH's human resources division has yet to implement this.

#### **Local administration**

The administration of tele-education within the University has changed over the years. Initially,the medical school had a computer committee that reported to the Faculty Administrative Committee. This committee met regularly with the University IT Division and co-ordinated infrastructure installation and the development of wireless links between teaching hospitals. The Department of TeleHealth was formed in 2002 as an academic department, which also provides support for tele-education and telemedicine. The head of department and a newly appointed Faculty IT Multimedia Manager reported directly to the Dean of the faculty. Two technicians support the IT Multimedia Manager and all work out of the Department of TeleHealth. After restructuring within the university, the administration of tele-education now rests with the Faculty IT Multimedia Manager.

## **Fogarty International Training Grant Model and International Tele-education**

The University of Natal was the local partner with Tufts University in Boston in an international training grant to educate people in sub-Saharan Africa in medical informatics and research in this field, which started in 1999. The program followed the then standard model of sending students and staff to the US for specific courses or to complete masters degrees and bringing faculty from the US to provide short courses. While meeting the expectations of the funders, those who were successful returned to unsupportive work environments and found themselves either working in isolation or unable to use their new knowledge. After the first cycle of funding the model was refined.

A postgraduate Medical Informatics program was established at the newly merged UKZ-N. Instead of sending students to the United States two or three students from a country were brought to South Africa for their postgraduate education. This enabled more students to benefit from the program but still took productive workers out of the workforce in their country. In the third cycle of funding, the model was further refined to take advantage of advances in ICT and improving bandwidth in African countries. Students were enrolled in several African countries and undertake their studies part-time while staying and working at home. A further aspect of the program is to develop capacity among staff within local universities in the field of medical informatics within local Universities so that in time they will be able to participate in teaching courses on the program with the end goal being to ultimately establish Medical Informatics Departments in their home institutions – "building the capacity to build capacity" (39). The program looks to educate 100 people across Africa.

This changed the approach to tele-education. No longer was teaching taking place by VC within KZ-N or to sites outside the Province via ISDN links. Attempts were made to convert ISDN input and output to IP using commercially available converters. This was not successful. A mixed mode approach to connectivity was needed. The first step was to adopt a learning management system to facilitate among other things, distribution of teaching materials, return and marking of assignments, interactive discussions and delivery, and return of examination papers and scripts. As a developing world country, it was decided to follow the Government's policy of using an open source software. After reviewing the literature, Moodle® was chosen and later adopted for use throughout the University.

An open source, IP, web-based VC was required to enable participation by students at sites that did not have ISDN connectivity. While Skype® had evolved to a stable one to one link, options like NetMeeting were not yet available. DimDim®,a product developed in India offered multisite IP based VC in a Skype® like environment. This free, open source solution integrated with Moodle® allowing scheduling within Moodle®. However, it required all connections to go through its server in India. For multisite connections from South Africa, a latency of up to 7 s was noted. There was also a differential latency between audio and video such that audio transmission was faster than video and when lecturing and changing a PowerPoint slide the audio preceded the video to the point that that the sound could be several slides ahead of what was seen. A commercial license was obtained to mount the DimDim® software on a local server, but this proved to be unstable and was abandoned.

The approach taken within the international training grant was to build capacity with students situated locally and outside of South Africa. This meant that there were students who were physically present at lectures and seminars in Durban, students at other sites in South Africa, and students at several sites in Africa. The problem was how to enable screen sharing and interactive audio communication between sites connecting using ISDN based and IP communication. The solution was simply to run the two systems in parallel. One computer was linked to the ISDN based videoconference unit. A second computer ran Skype® with a multisite license over the Internet with students at distant sites initially connecting through their local service provider who connected via satellite.

The initial experience was with the Kigali Institute for Health in Rwanda. At the far site students gathered in a venue with an Internet connection and linked their computer to a data projector so that they could see the presenter on a shared screen. Audio was amplified through desktop speakers, one of the students controlled a Webcam, aiming it at whoever was speaking and a cheap off the shelf and an omni-directional microphone on a long lead was passed around between participants as and when required. This approach required the presenter to remember that a dual system was in use.

For the people at the different sites to hear and communicate with each other,judicious placement of speakers,and microphones on a table in front of the presenter was required, so that audio from the Skype® participants was directed to the microphone of the ISDN participants and vice versa. Surprisingly, audio feedback was rarely encountered. Skype® proved to be a simple solution and as it developed into multi-conferencing, it was used to connect several sites. Skype® chat was also used to allow people to pose questions during presentations. Another solution available at the time but not used was Elluminate® as this requires a commercial license and our approach was to use open source or shareware solutions when and where-ever possible.

Screen sharing of presentations, demonstrations of software, and coding exercises presented another problem because of latency when using Skype. Different approaches were taken. One solution was to save PowerPoint presentations to Moodle® and have the presentation shown locally via a data projector to the assembled students. With limited bandwidth and high costs of data download, files of over five Meg proved difficult and costly to download. An alternative was to save the PowerPoint presentation as a handout with six slides to a page and save this in Adobe pdf format,

**Table 1 | Design-reality gap assessment showing the question asked, score, and answer**.

which substantially reduces the file size. One of the students at a far site would show the appropriate slide locally using Adobe Reader.

Another use of VC that was not envisaged is its use for invigilating examinations.

A VC bridge was acquired by the Medical School through another Fogarty International Training Grant (MEPI). This enables up to 48 sites to be connected simultaneously but only through the Internet. To include ISDN sites, one of the multisite VC units has to connect to the bridge via IP and then connect to the ISDN sites. The KZ-N DOH has also acquired a VC bridge, which has not been used for tele-education.

## **Reflection**

The local tele-education service and international program has evolved over time and adapted to changing needs, circumstances, and technology. It has grown and appears to be sustainable. When viewed in terms of current thinking about planning of eHealth projects few of the processes that are currently felt to be correct were undertaken.

It is said that there should be an eHealth strategy in place for successful introduction of eHealth programs (54). While the country had a telemedicine strategy, the telemedicine program had failed. Neither the University nor the KZ-N DOH had a


**www.frontiersin.org** November 2014 | Volume 2 | Article 173 | **41**

tele-education strategy. The University did have a strategy for the use of VC for administrative functions and meetings between its various campuses.

The problem of providing teaching to off-site trainees was well known and no formal needs assessment was made. A comparison of possible solutions was not made and the decision to use VC was made unilaterally based on circumstantial availability of VC equipment and connectivity. No assessment of human and financial resources was made. The decision to adopt the principles of no-one doing any extra work and starting small and learning form experience reduced human resource issues to technical and administrative support with the University providing the support. No formal business plan was developed. At the outset the KZ-N DOH supplied equipment that was not being used so no direct cost was incurred and agreed to pay for the ISDN line rentals and the call costs for the educational sessions. The University provided seed funding to set up a VC site at the medical school and provided the time of the technical staff. The regulatory environment was not considered to be an issue as there were no regulations relating to telemedicine or tele-education.

An eReadiness assessment was not undertaken as the infrastructure was in place and it was acknowledged that training on the use of the equipment would be required. A shortcoming at this stage was that while trainees were made aware of the services, hospital managers did not necessarily buy-in to the project. A fundamental implementation plan that included a change management plan was developed for each new site that was used. Growth of the number of programs participating was due largely to observation of other departments doing it. Although not formalized, each of the programmes participating in the first five years was evaluated at various times for user satisfaction and technical issues. No formal economic assessment has been made although rudimentary assessment was made of psychiatry teaching (43).

#### **DESIGN-REALITY GAP ASSESSMENT**

No risk assessment was made during the planning phase. This has been undertaken now, after the fact, in an attempt to assess the risk at the time. Design-reality gap assessment that evaluates seven dimensions, information, technology, processes, objectives and values, staffing and skills, management systems and structures, and other resources has been used as it is a relatively simple method (21). The seven dimensions are scored from 0 to 10 with zero reflecting no change between the design proposal and current reality (**Table 1**).

The score of 28 places it on the border line between a project that might be a partial failure and an e-health project that might fail totally, or might well be a partial failure unless action is taken to close design-reality gaps. A score of 14 or less suggests success. Despite the seeming lack of planning for the tele-education projects, the risks were not perhaps as great as might be expected.

## **CONCLUSION**

Tele-education has been used in different forms in the health sector in South Africa for a number of years. There is little hard evidence of its educational merit or economic worth. What is apparent is that it improves access to education and training in resource constrained settings. The development of local and international tele-education at the UKZ-N has not followed what is currently considered to be best practice but shows how programs can develop if there is a real need and the solution assists in meeting the need. Further work is required to analyze the economics of these tele-education endeavors.

## **ACKNOWLEDGMENTS**

This work was supported in part by National Institutes of Health Fogarty International Centre (grant number 5D43TW007004-11)*.*

## **REFERENCES**


**Conflict of Interest Statement:** The institution received payment from the Global Partnership Program for the development of four Health Informatics Building Blocks.

*Received: 15 June 2014; accepted: 18 September 2014; published online: 03 November 2014.*

*Citation: Mars M (2014) Tele-education in South Africa. Front. Public Health 2:173. doi: 10.3389/fpubh.2014.00173*

*This article was submitted to Public Health Education and Promotion, a section of the journal Frontiers in Public Health.*

*Copyright © 2014 Mars. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

#### **Ignacio Prieto-Egido<sup>1</sup>\*, Javier Simó-Reigadas <sup>1</sup> , Leopoldo Liñán-Benítez <sup>2</sup> , Víctor García-Giganto<sup>3</sup> and Andrés Martínez-Fernández <sup>1</sup>**

1 ICT Department, Rey Juan Carlos University, Fuenlabrada, Spain

<sup>2</sup> Rural Telecommunications Groups, Pontifical Catholic University of Peru, Lima, Peru

<sup>3</sup> EHAS Foundation, Madrid, Spain

#### **Edited by:**

Richard Wootton, University Hospital of North Norway, Norway

#### **Reviewed by:**

Aldo Von Wangenheim, UFSC – Federal University of Santa Catarina, Brazil Rodrigo Pires Dos Santos, Hospital de Clínicas de Porto Alegre, Brazil

#### **\*Correspondence:**

Ignacio Prieto-Egido, ICT Department, E.T.S. de Telecomuniación, Rey Juan Carlos University, Camino del Molino s/n, Fuenlabrada 28943, Spain e-mail: ignacio.prieto@urjc.es

Rural areas in developing countries are characterized by lack of resources, low population density, and scarcity of communications infrastructure. These circumstances make it difficult to provide appropriate health-care services. This paper explains research results achieved by Enlace Hispano Americano de Salud – Hispano American Health Link (EHAS) and how they have contributed to improve healthcare in isolated areas of developing countries through the use of information and communication technologies (ICT). As the first step, EHAS always collaborates with public health systems to identify its communication and information needs. Based on the analysis of needs, EHAS does research on appropriate technologies to provide communication in each context and on information systems suited to needs of health personnel. In parallel, EHAS has worked to provide applications that, making use of the communications services installed, could improve the health-care services in these remote areas. In this line, solutions to improve epidemiological surveillance or to provide telemedicine services (like a digital stethoscope or a tele-microscopy system) have been developed. EHAS has also performed several researches trying to ensure the sustainability of their solutions and has summarized them in a Management Framework for Sustainable e-Healthcare Provision. Finally, the effort to spread acquired knowledge has crystallized in a book that details all the technologies and procedures previously mentioned.

**Keywords: telemedicine, e-health, tele-stethoscopy, telemicroscopy, rural areas, ict4d**

## **INTRODUCTION**

Healthcare in developing countries is not equally accessible to people living in urban or semi-urban areas and those living isolated rural areas. Rural areas, like the one shown in **Figure 1**,face important challenges such as the lack of resources (trained professionals, equipped establishments, power supply, etc.), high population dispersion, and the scarcity of communications infrastructure (roads, public transport, telecommunications, etc.). These circumstances make difficult to provide an appropriate health-care service to the population living in these areas. Therefore, it is precisely in this context where information and communication technologies (ICT) can make the difference. This is the main goal of Enlace Hispano-Americano de Salud (EHAS) Foundation; improving health-care services in isolated rural areas of developing countries, through an appropriate design and use of ICT. EHAS is a research and non-profit institution. Its board of directors is composed mainly by Spanish and Latin American universities (Technical University of Madrid, Rey Juan Carlos University, Pontifical Catholic University of Peru, and Cauca University in Colombia).

The work of EHAS began at the end of the 1990s by analyzing the communication and information needs in rural primary healthcare in developing countries. The primary health-care system in Latin America is usually composed by two types of facilities: health centers and health posts. Health posts are usually located

in small communities and headed by health technicians that have received only a basic training. When a patient cannot be treated in the health post, he should be transferred to the referral health center,which is headed by physicians and have some infrastructure and equipment for diagnostic tests. In rural isolated areas, the trip from the health post to the health center can take 10 h in average and in urgent cases it can have a cost up to 2,000 USD. The last element of the primary health-care system is the hospital, where the patient can receive specialized medical attention. The analysis of information needs in this context started with a revision of existing studies on this matter (1–4), and continued by performing field researches in Peru and Nicaragua (5). The conclusions of that work showed that the main information needs in those areas where related to:


qualification (physicians, obstetricians, and nurses) prefers to move toward urban areas.


Another important fact is that voice communication is considered by different studies as the most important service in rural areas of developing countries (6). These information needs could be addressed by different telemedicine solutions, but there are several barriers to the deployment of new technologies in these scenarios: lack of infrastructure and electricity, low purchase power of the health establishments, and high maintenance costs due to the great distances and the scarcity of trained people.

## **COMMUNICATION NETWORKS FOR ISOLATED AREAS**

Commercial communication solutions were not designed for these environments because they do not take into account its technical and economical constrictions. After the mentioned initial studies, EHAS started to do research on two lines: appropriate technologies to provide communication in this context and information systems suited to the health personnel needs. One of the first steps in this process was to identify the requirements that a solution must fulfill to be considered sustainable: be very robust to reduce travels for maintenance, have low operation costs, and low power consumption because they will be powered using solar energy.

In order to face this communication challenge, wireless technologies appeared the most suitable ones. Most of the information needs could be addressed by asynchronous (off-line) systems with low-data transmission rates, such as email. Following this guidelines, two prototypes were designed focusing on the needs of rural

**FIGURE 1 | Typical community in an isolated area without infrastructures**.

staff. These prototypes were based on open technologies (hardware and software) in order to facilitate appropriation, and reduce acquisition, development, and maintenance costs. They were based on very high frequency (VHF) and high frequency (HF) radios to provide voice and low-data rate services over long-distance links. The prototypes used the sound card of a personal computer (PC) to modulate with the digital information (such as emails) an analog sound signal and send it through the audio channel provided by the VHF or HF radio. Using these prototypes, two pilots were deployed to communicate more than 75 isolated health posts in Colombia (7) and Peru (8), like the one shown in **Figure 2**.

The pilots served to measure the reliability of the technology and its impact on the health system (both in the staff and in the health-care). After 9 months of operation in 39 health posts of Peru, evaluation results showed that the mean consultation rate per facility was increased from 3 per month (95% CI 1.5–4.5) to 23 per month (95% CI 14.7–31.5). Moreover, there were 205 emergency transfers and the system was used in all cases to coordinate (as shown in **Figure 3**) and alert the referral center. Finally, the

**FIGURE 2 | Infrastructure of solar panels and tower installed in an isolated health posts**.

**FIGURE 3 | Health technician using the VHF system to coordinate an emergency**.

mean time required to evacuate a patient was reduced from 8.6 to 5.2 h (8). This field research in Peru was conducted with the help of the Rural Telecommunications Group of the Pontifical Catholic University of Peru.

Although the impact of these technologies proved to be very important, the bandwidth provided was still quite low (17 and 3 kbps with VHF and HF, respectively) and it was not possible to use them to transfer images or videos. At that time, some works (9, 10) had remarked the strategic role that IEEE 802.11 (the WiFi standard) could play for deploying low-cost networks in such scenarios. Besides,WiFi devices had spread and become very popular (due to its low cost) all over the world at that point, and EHAS decided to test if it was possible to use that technology to provide high transmission rates over long-distance links. The result to this question was positive (11), and some adjustments in the WiFi protocol (specifically in the ACKTimeout, CTSTimeout, SlotTime, and CWmin parameters) were proposed to improve significantly its performance over long distances. Learning from previous experiences like (12) and (13), this technology was used to deploy Wi-Fi for Long-Distances (WiLD) networks in Cuzco and Iquitos (14), both in Peru. Commercial embedded computers with a Linux Voyage distribution were used as WiFi transmission systems (15), and new antennas (suitable for WiFi frequencies) were installed in the towers previously used for VHF/HF systems. These networks were employed to provide three basic services: internal telephony, email, and Internet access. WiFi networks were not devised initially to support voice services, but voice over IP (VoIP) allows providing this kind of service like shown in **Figure 4**. Thanks to it, a telephony system was installed to allow health establishments to communicate among them without paying for calling time, and to call to the public telephone network using prepaid cards.

For the deployment of these pilots, it was not only necessary to develop the prototype but it was also required to do research on how to adapt the deployment to context conditions. The lack of electricity was solved by developing an autonomous wireless node for WiLD networks powered through solar energy (15) with 75W

**FIGURE 4 | Health technician using the VoIP system**.

panels and 12V batteries. It was also necessary to design the system considering environmental characteristics such as tree height, humidity, and thunderstorms in the Amazon jungle, or mountain altitude and ground conductivity in the Andean region. Approximately, the throughput in these networks was limited to 3 Mbps, so some research on how to optimize resources and maintain the quality of main services (QoS) was also performed (16).

In the last years, EHAS has continued its quest for innovative communication solutions for rural isolated areas, studding the latest versions of 802.11 standard and new technologies such as IEEE 802.16 (known as Wimax) (17). This work has served to improve existing networks and use them to provide services requiring more network resources, like the telemedicine services that will be explained in the next section.

Nowadays, communication systems are converging toward IP and mobile technologies, and EHAS is working to bridge another important aspect of the digital divide: the access to mobile services in rural isolated areas of developing countries. Mobile telephony and Internet access bring the opportunity to break isolation by reducing barriers to knowledge acquisition and participation, and by allowing these communities to access new economic opportunities (18, 19). Different studies have shown that the contribution of mobile telephony to economic development is greater in less developed economies, and that this technology impacts directly on the reduction of poverty levels. Specifically, a study of the World Bank proved that a 10% increase in bandwidth penetration can cause a 1.4% rise of the gross domestic product (GDP) (20). However, isolated areas are not profitable for mobile operators due to the low density population and the high cost of infrastructures. To face this problem, EHAS is participating (together with 10 more partners) in a research project known as TUCAN3G that is being funded by the European Commission through the FP7 (21). Its objective is to prove that it is possible to use low-cost 3G station bases designed for indoor scenarios (known as femtocells) to provide mobile voice and data services (3G) in isolated areas of developing countries, using a heterogeneous WiFi/WiMAX-VSAT network as backhaul. The research is developing solutions to adapt and integrate the different technologies used, and is also trying to develop a sustainable business model in order to involve mobile operators in the solution. The participation of operators is required because they have the licenses to operate in the mobile bands. If this solution proves feasible, it could help to cover communication needs of the primary health system in rural isolates areas.

## **TELEMEDICINE SERVICES**

In parallel with the work on communication technologies explained in the previous section, EHAS has done research to develop applications that, making use of the communications networks installed, could improve both the health information system and the health-care services in these remote areas. The first approaches in this line focused on increasing the efficiency of the epidemiological surveillance system and allowing distance training in remote areas (22) using voice and low-rate data systems based on VHF.

The need to improve health information systems is related with the existence of prevalent diseases such as dengue, malaria, infectious respiratory diseases, or diarrhea diseases. Having updated and accurate information of a disease situation is crucial to prevent or promptly face an epidemic. When EHAS started its work, most of the existing research was oriented to the analysis of data, although the main problems in the rural areas were related with the collection process. Therefore, EHAS contribution was to develop an appropriate solution to collect this information using low-cost and open-source technologies designed for these specific contexts. The Telematics Department of the University of Cauca developed a system for collecting, sending, processing, visualization, and feedback of epidemiologic information. This system was designed for scaling at national level, but it was previously tested in a pilot area, where it solved the problem of subregister, increasing the volume of data collection a 15%. A similar approach was used to design a new distance training system, given that existing solutions were based on video transmission (which requires a high a bandwidth that was not available) or required permanent connection to Internet (which was hard to guarantee in remote locations). A distance training system synchronized through email messages was developed by the Technical University of Madrid and the Carlos III University of Madrid. Both systems were based on a combination of email and XML (eXtensible Markup Language) technologies and were designed to work over slow, unreliable, and asynchronous connections. These solutions were used to offer remote courses about "child diarrhea," "nursery attention in the primary health-care pre-emergency services,""epidemiologic surveillance," and "health education and disease prevention, grass-root level oriented" in 52 health establishments of Dominican Republic, Cusco (Peru), Cauca (Colombia), and Guantanamo (Cuba).

The research on health information systems has continued since then and nowadays is based on the use of open-source tools such as OpenMRS (23) or DHIS2 (24). The advantages of using these tools in concrete countries, such as Peru, Paraguay, Mexico, or Colombia have been in deep studies, and currently DHIS2 is being deployed in some regions of Colombia (25). When the communications technologies started to provide high bandwidth networks (taking advantage of WiLD solutions) EHAS started to work on ways to improve the diagnosis capabilities of the rural establishments. The basic idea was to allow doctors to remotely support health technicians using real-time communications tools based on voice and video. In these scenarios, a remote consultation has great advantages:


One of the first works in this line was a digital stethoscope (26) aimed to diagnose infectious respiratory diseases, which are the main infant mortality cause in rural areas of developing countries. When this digital stethoscope was developed there was no other device with similar performance, and nowadays there are solutions that send the sound in real time but do not offer a video of the patient. This digital stethoscope allows a doctor to support a health technician in the diagnosis process of respiratory or cardiac pathologies. It sends the audio and video in real time, so the remote doctor is able to know the position of the stethoscope associated with each sound, to ask the technician to change this position and to give instructions to the patient. This information is crucial for the doctor to make a remote diagnosis. The stethoscope also allows recording the sounds in order to send them in a file to ask for a second opinion. The information to build the digital stethoscope is shared under GPLv3 license (for free and open-source software). Its design is simple in order to make it possible to produce it in universities or small companies of developing countries. But at the same time, the design is robust enough to be used in areas with extreme climate conditions (high temperatures, high humidity rate, etc.). Moreover, the required components are low cost (approximately 200\$) in order to avoid that the cost become a barrier for its use. All this characteristics has been thought to develop a suitable digital stethoscope shown in **Figure 5** that can be used in rural areas of developing countries.

A microscopy would be a very complementary tool for the digital stethoscope because "Pneumonia, diarrhea, and malaria together killed roughly 2.2 million children under age five in 2012" (27). However, health technicians do not have the knowledge to diagnose using the microscopy. To face this problem, EHAS is currently working in two lines in a project funded by the Spanish Agency for International Development Cooperation (AECID). First, EHAS is designing and evaluating very simple protocols to prepare microscopy samples with the resources available in remote health posts. Secondly, EHAS is developing a low-cost telemicroscopy system based on on-board computers (28) shown in **Figure 6**. The idea is to teach health technicians to prepare the microscopy samples and provide them with a tele-microscopy system. In this way, they would be able to send the microscopy images in real time to a specialist in order to get a diagnosis. Other two prevalent diseases in developing countries are included in

**FIGURE 5 | Doctor using the digital stethoscope developed by EHAS Foundation**.

this research: tuberculosis and cervical cancer. It is important to emphasize that both the protocols and the tele-microscopy system are being evaluated by specialist in order to guarantee that this solution provides a sensibility and specificity equivalent to a traditional diagnose.

These telemedicine services are being used in the WILD network deployed in Iquitos, which is known as the Napo Network and that communicates 11 health posts with its reference health center and with the regional hospital. The project to deploy these services in this network started in 2010 by improving the communication and the information protocols. However, the telemedicine systems were not installed until 2013. The use of the networks is nowadays very intensive, the evaluation on the use of this equipment is still being performed, but the mean of VoIP call per health establishment is 2,000 calls per month (29). To evaluate the impact of these initiatives some indicators of the Napo Network are being compared with a control group. The selected control groups are the health establishments of the Tamshiyacu health network, that shows a high Pearson coefficient (*r* = 0,818), and has a similar demographic and isolation conditions as the Napo Network. The impact on health indicators is hard to measure and is still being evaluated, but the impact on processes has already been analyzed. **Figure 7** shows the impact on epidemiologic silence (the percentage of health reports sent by the health establishment that get lost and do not reach the Epidemiologic Department of the Regional Government), showing a clear reduction in the Napo Network since 2010. The project also had an important influence on the exchange on information about the number of attentions that were notified to the Health Insurance System (SIS) of Peru (**Figure 8**). The number of notifications grows a 50%, and thanks to that the income of the health network (coming from the national Health Insurance System) increased an 80%.

Enlace Hispano-Americano de Salud is now performing a research project to reduce maternal mortality in rural areas of developing countries through the use of portable ultrasound scans

**before reaching the epidemiologic department**.

(like the one shown in **Figure 9**) and blood tests. These tools are basicfor pre-natal care, but they are unavailable in many rural areas of developing countries due to the lack of electricity and suitable technologies. EHAS is evaluating an innovative and cost-effective technology (a backpack containing a small solar panel and the required equipment for blood testing and ultrasound spot screening) to improve the access to and the quality of pre-natal care in these rural areas. Nurses are trained to locally use these tools and perform a basic check-up oriented to detect the most common obstetric complications. The information is stored in the computer offline and synchronized with a web platform when Internet connection is available. In this way, the information is later supervised by a specialist that confirms the diagnosis and improves the

nurses training. These tests could detect 80% of obstetric complications on time, with an average cost of about US\$ 25.00 per pregnant woman (including two ultrasounds, two blood, and two urine analysis). The aim is to convert an obstetric emergency (that means expensive transfers and attentions in many cases already useless) into a reference to an appropriate health center with a routine transfer 1 or 2 weeks before the labor. The first results obtained (with a sample of 1000 women) are very promising: the health mortality has been reduced to 0 and the newborn mortality has decreased a 50% (30).

## **SUSTAINABILITY**

The technical feasibility of a solution could be proved in a laboratory test bed; however, developing interventions are very complex and achieving sustainability is not a mere technological issue. That is why EHAS Foundation always includes a pilot deployment in its research projects, as a way to consider other factors involved in sustainability, such as economic, financial, institutional, educational, and social and cultural aspects (31). In this process, local universities have always played an important role as local partners due to several reasons:


Enlace Hispano-Americano de Salud has based on most of its solutions in open-source software in order to facilitate technology

**FIGURE 10 | Health technicians being trained in the use of health information systems**.

transfer and to encourage other institutions to replicate and improve the proposed solutions. TheVHF modem, theVoIP server, the health information systems, the digital stethoscope, and the tele-microscopy system previously explained are examples of this, but it also includes software to manage the networks (32), which is a very important issue in the network maintenance.

The economical factor is also studied in the pilots, proving that the net economic effect of the telemedicine program over a 4-year period was clearly positive, and that the additional operational costs introduced by the telemedicine system were lower than the savings produced for the health-care network (33). These savings were related to the reduction in patient referrals and the cost associated to them. Prior to the availability of the EHAS telemedicine system, there was a mean of 11.1 urgent patient referrals per year from the HPs and 14.0 referrals per year from the HCs. After the implementation of telemedicine, patient referrals fell to 2.5 (With a *P* value of 0.03) per year from the HPs and to 8.4 per year from the HCs (with a *P* value of 0.17).

In order to ensure sustainability and bring together all the experience related with sustainability, a Management Framework for Sustainable e-Healthcare Provision was developed (34). This framework considered human resources, logistics, and budget guaranteed for operation and maintenance of the network, and it was applied in the Napo Network. The network was transferred to the Regional Government, who is now responsible of the whole network and guarantees its continuity. This strategy implies an organizational change (definition of new roles, annual budget approval, etc.) that is also described in the framework.

Finally, the effort to spread the acquired knowledge has crystallized in several scientific papers and a book (35) that details all the technologies and procedures previously mentioned. This book aims to spread the use of ICT for health in rural areas of developing countries.

## **FUTURE WORKS**

Until now, the challenge was to find communication technologies to connect isolated health establishments and to provide services such as remote training, support to diagnose, or information exchange. Nevertheless, nowadays the mobile systems have reached a high penetration even in the rural areas, and telemedicine solutions are expected to converge with mobile networks. Therefore, future research should make compatible traditional medical instruments (stethoscope, microscope, ultrasound scanner, etc.) with mobile operation systems such as android or iOS. In this way, developing countries could take advantage of the products developed in the industrialized areas, something that until now has not been possible. This is, especially, relevant in countries with a middle human development index, which nowadays have resources to hire medical specialists but where it is hard to find specialist willing to work in rural areas. In this context, telepresence is revealing itself as a promising alternative. In a complementary way, research on solutions to provide an automatic diagnose will provide useful screening tools and will help to promptly detect and treat the most serious cases. Strategic alliances to combine different technologies are other key point where research is needed. Remote consultation systems could learn from initiatives like"Water First – Health Follows" of the e-Health-Point project, where prevention (access to drinkable water) is combined in the same monthly fee with medical attention (in case prevention fails). Finally, an important effort is required to include medical teleconsultation in developing countries legislations as part of the portfolio of services in order to make refundable this type of attentions. This is an important step to transform an isolated set of pilot projects into a part of national health-care systems.

## **CONCLUSION**

The research line of EHAS Foundation has focused during more than 15 years on designing, developing, and evaluating ICT applications to face health problems in rural areas of developing countries. It started developing communication solutions for areas that were completely isolated, and the evolution of the technology allowed EHAS to drift toward more ambitious applications such as telemedicine services. Some of the networks here described have been working for more than 7 years since they were transferred to local actors, what proves that the sustainability plans developed are reaching their objective. The tele-stethoscopy and

## tele-microscopy systems have proved their sensibility and specificity through clinical trials (still in publishing phase), and several pilots are being evaluated to prove that it is possible and efficient to provide telemedicine services in remote areas. Results show that the impact is positive for the patients (that receive an improved healthcare), for health personnel (that has more resources, more opportunities and more confidence), and the whole health system (that increases available information and the health networks income). These results have contributed to spread telemedicine opportunities and in this line other telemedicine networks have been deployed in Peru, Ecuador, and Colombia. The next challenge is to find mobile technologies to make more accessible the ICT solutions, a field where EHAS is already working. These research results can be useful to all kind of actors (national public administrations, multilateral institutions, industry, academy, civil society, etc.) in order to promote really relevant and sustainable solutions in telemedicine for rural regions of developing countries.

## **SUMMARY OF PROJECTS**

This section will provide a summary of the projects described in this paper, showing the most relevant figures of each one (**Table 1**).

## **PROJECT I**

EHAS – VHF Networks (7, 8, 22, 33). This Project was executed in Alto Amazonas (in 56 health establishments) and Datem del Marañon (in 17 health establishments), which are both provinces of the Loreto Department in Peru, and also in the districts of Silvia and Jambalo (in 21 more health establishments), which belong to the Cauca Department in Colombia. The funds were provided by the Andean Health Organization (ORAS) and the Spanish AECID. This project deployed communication systems for voice and data (for email access only) that were based on radio VHF links (using TCP/IP over AX.25). The first 39 Peruvian systems were installed in 2001 and the last 17 ones were installed in 2006, and all of them have been operating until 2012, date on which most of them were upgraded with WiFi over long-distance (WiLD) technology. The systems in Colombian were installed in 2004 and have been operating until 2010. Voice communications were used to query synchronous clinical questions (mostly related


#### **Table 1 | Summary of EHAS projects.**

to respiratory infections and diarrheal diseases) and to coordinate urgent transfers. The data system was used asynchronously to coordinate activities, as a tool for tele-training and also to send epidemiological reports. The average cost per installation was US \$5,500 per establishment (10% of annual maintenance included), which included a computer and a printer, a radio transceiver, towers and antennas for communication, a system for protecting against lightning, and a solar power supply solution. These systems have been operating an average of 7 years and have been used to attend more than 5,000 patients each, what implies an average cost of US \$2 per patient (taking into account CAPEX and OPEX). This project had a high impact in the reduction of travel of health staff and in the coordination of urgent transfers, as has been stated previously.

## **PROJECT II**

EHAS – Napo Network (11, 14, 15, 21, 29, 34). Project carried out in the basin of the Napo River, in the Maynas province, in Loreto, Peru. The Project deployed a telecommunications network with WiLD technology in order to connect 15 establishments of health along 450 km of river. The Project was initially funded by the ORAS in 2007, and was completed with funding from the Madrid City Council (Spain) in the year 2009 (the network is still operating nowadays). The initial equipment was composed by a VoIP telephone and by a computer with permanent access to the Internet and a videoconference software. The system has been used to perform synchronous consultations between the health technician and his reference doctor, and also between the doctor and the specialists of the regional hospital, particularly in pathologies related to obstetrics, pediatrics, and dermatology. The Internet connection has been intensely used to control the stock of medicines and send epidemiological reports. Video conferencing has been used for remote training of technicians, which reduces the number of trips, and therefore, the travel expenditures of the health system. The average cost per installation exceeded US \$20,000 due to the need for high towers to secure the line of sight between antennas. Assuming equal maintenance costs than in the previous project, and taking into account that each establishment has attended more than 8,500 patients, the average cost per patient is US \$4, without considering the effect of remote training and the improvement on information management. This project has a high impact in the reduction of urgent transfers, the reduction of epidemiological silence areas, and the maintenance of medicines stock.

## **PROJECT III**

EHAS – Tele-stethoscopy (26). Fifteen tele-stethoscopes were installed in the network of the Napo River in order to provide remote cardio-respiratory auscultation services. This project was funded by the Madrid City Council in 2011 and continues operating nowadays. The system allows reference doctors to remotely monitor cases of acute respiratory infections. To a lesser extent, the system is being used so that general practitioners can receive a second opinion from the specialists in cardiology at Iquitos Regional Hospital. The cost of each system is US \$600 per point (including installation costs) and has been used with more than 750 patients by establishment, which means a cost of less than a dollar per patient. Its impact on the reduction of morbidity and mortality has not been scientifically evaluated due to the lack of resources to study a randomized control group in such a remote area.

## **PROJECT IV**

EHAS – Tele-microscopy (28). Fifteen tele-microscopy systems have been installed the last year in the Napo network, and are used to make remote diagnosis of malaria, tuberculosis, parasitic infections, and cervical cancer. This project has been funded by the Spanish AECID. The equipment and installation costs do not exceed US \$500 and the estimated average usage is two daily cases by establishment.

## **PROJECT V**

Healthy Pregnancy (30). The pilot project was performed in Alta Verapaz, Guatemala, between 2012 and 2013. It was funded by the Polytechnic University of Madrid, and it has equipped three nursing brigades with a portable ultrasound kit, a folding solar panel (for powering), and a system for performing blood tests using dried blood. The system is used to detect early obstetric complications (malposition, placenta praevia, twins, preeclampsia, infections, anemia) and identify complicated deliveries that should not be carried out in rural areas lacking of medical supervision. One thousand pregnant women have been attended using this kit, and results show that neonatal mortality has been reduced a 65% and maternal mortality has disappeared in the sample. The cost of the kit is around US \$5500 and it allows attending an average of 400 pregnant women per year. A business model has been designed to scale the project, estimating a cost of \$25 per pregnant woman. Now, with funding from AECID, USAID, and IDB, the Project is trying to increase the sample to 10,000 pregnant women only in Guatemala.

## **ACKNOWLEDGMENTS**

This paper is based upon work funded by the Spanish Agency for International Development Cooperation (AECID) with the projects 11-PR1-0131 and 12-PR1-0438, by the Technical University of Madrid, and by the European Community in the framework of the FP7 project TUCAN3G IST-601102 STP. The authors would like to acknowledge the contributions of the colleagues from TUCAN3G Consortium (http://www.ict-tucan3g.eu). The content is solely the responsibility of "EHAS Foundation" and do not necessarily represent the opinion of the funders.

## **REFERENCES**


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 22 July 2014; paper pending published: 23 July 2014; accepted: 28 September 2014; published online: 15 October 2014.*

*Citation: Prieto-Egido I, Simó-Reigadas J, Liñán-Benítez L, García-Giganto V and Martínez-Fernández A (2014) Telemedicine networks of EHAS Foundation in Latin America. Front. Public Health 2:188. doi: 10.3389/fpubh.2014.00188*

*This article was submitted to Public Health Education and Promotion, a section of the journal Frontiers in Public Health.*

*Copyright © 2014 Prieto-Egido, Simó-Reigadas, Liñán-Benítez, García-Giganto and Martínez-Fernández. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

## Telemental health in the Middle East: overcoming the barriers

## **Hussam Jefee-Bahloul \***

Department of Psychiatry, Yale School of Medicine, New Haven, CT, USA \*Correspondence: hussam.jefeebahloul@gmail.com,hussam.bahloul@yale.edu

#### **Edited by:**

Richard Wootton, University Hospital of North Norway, Norway

**Reviewed by:**

Shridhar Murlidharrao Kadam, Public Health Foundation of India, India

#### **Keywords: telemental health, mental health services, Middle East, developing countries, telepsychiatry**

## **INTRODUCTION**

The Middle East (ME) is a heterogeneous part of the world with variations in mental health services. The spectrum includes countries that have well (e.g., Israel), fair (e.g., Turkey, Iran, Saudi Arabia), or poor (e.g., Iraq, Syria) services according to the data provided by Jacob et al. (1) (number of psychiatrists per 100,000 population: Israel, 13.7; Bahrain, 5; Qatar, 3.4; Kuwait, 3.1; Lebanon, 2; UAE, 2; Iran, 1.9; Oman, 1.4; Saudi Arabia, 1.1; Turkey, 1; Jordan, 1; Iraq, 0.7; Syrian, 0.5; Yemen, 0.5; compared to USA, 13.7; Canada, 12; UK, 11). Given the disparities in mental health services there is a need to implement telemental health (TMH) in the ME. This article will discuss barriers of TMH in the ME and propose recommendations for implementation based on published studies and the author's experience.

## **BARRIERS TO TELEMENTAL HEALTH IN THE MIDDLE EAST**

Telemental health (or telepsychiatry) is a form of telemedicine that provides mental health services via telecommunication instead of face-to-face. There are two main modalities in TMH: synchronous and store-and-forward (S&F). Those modalities can be delivered through different means of telecommunication (e.g., telephone, Internet). Videoconferencing is a "synchronous" and "real-time" interaction between patients and therapists via video connection. S&F telemedicine is a transmission of recorded clinical material between referring physicians and specialists in an "asynchronous" manner [see Ref. (2) for a recent review].

Western studies have found TMH to be cost (3) and clinically effective (2, 4) in increasing access to care. However, a

search of four engines (PubMed, Ovid, Google Scholar, and IMEMR) for various combinations of the following keywords: "telepsychiatry,""telemental health,""tele\*," "videocon\*," "web-based," and "Internetbased,"cross-matching a list of all the countries in the ME was done. Publications that reported on, or addressed TMH in the ME were included [all publications in English and one Farsi paper (5)]. The search found 11 publications [10 published (5–14) and 1 in-press (15)] (**Table 1**). A full systematic literature review of TMH reports in the ME and North Africa region, which is beyond the scope of this opinion article, is prepared to be published elsewhere. Only three published reports are of randomized controlled trials (RCTs); which were done in Turkey, Israel, and Iraq (9, 10, 12).

Four main barriers to implement TMH in the ME are frequently observed: cultural, technical, financial, and regulatory.

## **CULTURAL BARRIERS**

In cultures where interpersonal relationships are valued and direct doctor–patient interaction is expected (16), barriers to use the technology in delivering medical services would naturally arise. Two categories of cultural barriers will be presented here.

## **PATIENT RELATED BARRIERS**

Frequently, patients in the ME voice concerns about physician's background and culture, technology itself, security, and privacy during encounters (17). In a study conducted by author and colleagues, Syrian refugees were asked about their acceptance of mental and TMH (PASSPORT study) (15). Hesitance toward the use of technology was observed for reasons such as: privacy, distortions to the doctor–patient

relationship, and unfamiliarity with the technology (15).

Age, gender, and religion are important cultural factors. Multiple studies have showed that elderly patients are less likely to accept TMH compared to younger population (14, 16, 18, 19). In general, females are less satisfied with, and less likely to see, a psychiatrist in the ME (20). Females in the Islamic tradition tend to be more conservative than males and have less public interactions, which may reflect their hesitance toward mental health services (20). Even though some studies found no difference in acceptance of TMH between genders, females in the PASSPORT study were more likely to accept face-to-face psychiatry (*P* < 0.05) and less likely to accept TMH (*P* = 0.64) compared to males.

Religion was not recorded as a variable in the PASSPORT study; however, the majority of the refugees were of the Northern – more conservative – part of Syria. The factor of religion in acceptance of TMH was noted in an Israeli survey of patient's attitudes and willingness toward using TMH. The study showed that religious patients were less likely to accept TMH when compared to those who are secular (14).

## **PROVIDER-RELATED BARRIERS**

There are no reports examining provider-related barriers to TMH in the ME, however, the lack of knowledge or experience in the technology and the need for training are expected barriers for telemedicine (17). In addition, some providers associate the use of technology with inefficiency (e.g., diagnosing might take longer), loss of revenue (fixing technology malfunctions would waste time), and remuneration difficulties (e.g., most patients pay out-of-pocket) (17).


**Table 1 | Published**

 **studies of telemental**

 **health in the Middle East**. BPRS, brief psychiatric rating scale; CGI, clinical global impression scale; VC, videoconferencing.

## **CULTURAL FACILITATORS**

Overcoming the cultural barriers requires strategies for training providers and increases their exposure to the technology and their awareness of its applicability. On the other hand, increasing public awareness of the use and effectiveness of technology may help facilitate patients' acceptance. In a Swedish study, Middle Eastern population who experienced TMH services (in Sweden and Denmark) reported high level of satisfaction (21), which can indicate a possibility of cultural acceptance with experiencing the technology.

## **INFRASTRUCTURE AND TECHNICAL BARRIERS**

While some countries in the ME have an infrastructure that might allow implementation of telemedicine (e.g., Israel, Turkey, Jordan, and UAE), others do not (e.g., Syria, Iraq, and Iran). Some clear examples of existing infrastructure barriers include electricity and Internet. Electricity for example is a precious commodity in some countries and obviously is a prerequisite need for telemedicine and TMH [even though; mobile technology can overcome the unreliability of electricity (22)]. High bandwidth capacity is another important requirement that would guarantee clarity of picture and voice, as these elements are crucial for the therapeutic relationship (23). The availability of trained technical support personnel and medical support (for emergencies) in remote settings represents another obstacle in areas suffering from shortage of technical and medical services to start with.

## **INFRASTRUCTURAL AND TECHNICAL FACILITATORS**

Effectiveness pilot studies are needed to make a political case and draw investment from local governments. A framework was created by Alajlani et al. (24) to help assess ME countries' readiness for implementation of telemedicine. The framework was based on stakeholders' interviews and surveys of physicians in both Syria and Jordan in 2010 (prior to the Syrian civil war). The framework included recommendations for institutions or agencies aiming at building telemedicine systems in the ME. Some of these recommendations were studying countries' changing policies, ensuring availability of human/technical resources, intensive training of staff, efficient models

of fee exchange, and creating partnerships with stakeholders and local organizations [see (17, 24) for the framework]. Although this framework was based on work in only two countries, the authors concluded that this framework is applicable in most countries in the ME.

## **LEGAL AND ETHICAL BARRIERS**

The legal system is often driven by different philosophies of health care and usually varies among different world regions (25). As telemedicine is not an integrated part of health systems in the ME, there are no regulations or legislations that guide such practices. Medico-legal issues, licensing requirements, regulation, and quality assurance issues would naturally arise soon after such implementation. In addition, confidentiality, data protection, and patient privacy are main issues in TMH. Fear of political persecution is a threat unique to, and commonly seen in, the ME. This fear can impact patients' acceptance to the technology (15), and hence require implementation of policies to protect information transmitted electronically. In addition, informed consent and availability of a system that allows patients' full access to their medical records should be thought of when designing TMH systems (25).

## **LEGAL AND ETHICAL FACILITATORS**

The process of building a TMH in the ME would require a parallel re-construction of the medico-legal system to allow for more considerate approach to sensitive ethical concepts such as confidentiality, informed consent, and liability.

## **FINANCIAL BARRIERS**

While funding is not believed to be a barrier in some countries (e.g., Qatar, Kuwait, and UAE), it is a major obstacle in others (e.g., Jordan and Syria) (17). Private hospitals might pioneer telemedicine projects and implement the technology faster than public hospitals, given the little governmental and public interest (17). While cost effectiveness of TMH has been established in the West (3), doubts about it in the ME do still exist. An Israeli RCT had conducted a cost analysis of a TMH service in Israel and found videoconferencing to be more expensive compared to face-to-face encounters (223% more expensive when adding hospitalization costs, 32% without

and 10% with inclusion of patient travel costs) (9).

## **FINANCIAL FACILITATORS**

Until more studies demonstrate cost effectiveness of TMH in the ME; convincing investors and stakeholders to invest would remain difficult. Adopting S&F applications might be less expensive; however, this needs to be demonstrated by cost– effectiveness comparison studies in the ME.

## **BEYOND THE BARRIERS**

These barriers are not isolated entities to be overcome separately. They are countryspecific intertwined and fixed complexes that act synergistically. For example, the provider-related "cultural" hesitance is believed to be due to "financial" concerns and lack of faith with the governmental "infrastructure." Providing intensive training and education for physicians about TMH without adequate implementation of an electronic fee exchange system and governmental allocation of supportive resources would not allow for active physician participation. Another example is the relationship between patient related "cultural" hesitance and the status of the legal and ethical bodies in a given country. Patients may choose not to participate in TMH due to lack of faith in the protective laws implemented by governments. Beyond the availability of proper protective laws, the actual governmental enforcement and policing of such laws is mandatory to increase patients' comfort. Addressing these intertwined schemas of barriers requires a reconstructive process with involvement of policy makers, government, and agencies of interest.

## **ANECDOTAL EXPERIENCE AND FUTURE DIRECTIONS**

In personal experience, patient interaction via TMH has not been feasible on a systematic scale in the ME. This is given the unreliable quality of telecommunication in the ME, the lack of clear medico-legal legislations for TMH, and the palpable hesitance toward it among both providers and patients. Notwithstanding these barriers, fruitful clinical supervision for a Syrian psychiatrist in Jordan was reported in a pilot project using Skype technology (8). Notably, the quality of supervision was highly affected by inadequate Internet connection in Jordan. Participants in videoconferencing often felt unable to participate on equal footing due to imperfect timing in verbal exchange. This is believed to have hindered the ability to build strong rapport and work alliance. Also, lack of financial compensation was a significant barrier to sustaining the work. In another (ongoing) project, mental health training of educated young Syrian refugees in Turkey who are involved in providing psychological support to fellow Syrian refugees is taking place via videoconferencing. Of note, the Internet connection in Turkey is allowing a better quality videoconferencing with uninterrupted training sessions. Another project that links referring physicians in the field to psychiatry specialists all around the globe is underway (Syrian Telepsychiatric Network "STPNe"). This project includes a TMH consultative service that uses text- or audio-video-based S&F modality that allows specialists to provide clinical advice on presented clinical cases.

## **CONCLUSION**

Implementing TMH services in the ME is a step that can bridge a substantial mental health gap. It can help as clinical or educational mean through building the capacity of mental health workers. TMH is a viable solution to increase access of quality mental health services in the ME, however, its implementation is facing multiple barriers. The ME is a heterogeneous region of the world with a spectrum of stability that ranges from stable, unstable but not in conflict, to conflict setting. Countries in the ME also vary in their readiness to adopt the technology and built TMH systems. Approaches to build these systems should be based on preliminary assessment of the specific cultural, financial, legal, and infrastructural need of each country (24). It is worth noting that the countries with better ability to adopt TMH are those with a smaller mental health gap. In countries with the greater gap such as Syria, Iraq, and Iran; it is the author's opinion that shifting efforts from videoconferencing to less-bandwidth-demanding modules (S&F or web-based interventions) might be a feasible first step.

## **ACKNOWLEDGMENTS**

The author acknowledges Dr. Andres Barkil-Oteo as a collaborator in the anecdotal telemental health experiences outlined above. The author also acknowledges Dr. Kimberly Ann Yonkers for her help in preparing this manuscript.

## **REFERENCES**


to referral for telepsychiatry (PASSPORT Study). *Telemed J E Health* (In Press).


**Conflict of Interest Statement:** The author declares that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 10 May 2014; accepted: 03 July 2014; published online: 17 July 2014.*

*Citation: Jefee-Bahloul H (2014) Telemental health in the Middle East: overcoming the barriers. Front. Public Health 2:86. doi: 10.3389/fpubh.2014.00086*

*This article was submitted to Public Health Education and Promotion, a section of the journal Frontiers in Public Health.*

*Copyright © 2014 Jefee-Bahloul. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

## A low-cost tele-imaging platform for developing countries

**Kokou Adambounou1,2\*,Victor Adjenou<sup>1</sup> , Alex P. Salam<sup>3</sup> , Fabien Farin<sup>2</sup> , Koffi Gilbert N'Dakena<sup>1</sup> , Messanvi Gbeassor <sup>1</sup> and Philippe Arbeille<sup>2</sup>**

<sup>1</sup> Unité de Télémédecine du Centre Hospitalier Universitaire Campus, Université de Lomé, Lomé, Togo

<sup>2</sup> Unité de Médecine et Physiologie Spatiales (UMPS-CERCOM), Centre Hospitalier Universitaire Trousseau de Tours, Tours, France

<sup>3</sup> Chelsea and Westminster Hopital, London, UK

#### **Edited by:**

Richard Wootton, University Hospital of North Norway, Norway

#### **Reviewed by:**

Jui-chien Hsieh, Yuan Ze University, Taiwan Tetsuya Miyashita, Yokohama City University Hospital, Japan

## **\*Correspondence:**

Kokou Adambounou, 05 BP 633, Unité de Télémédecine du CHU Campus, Université de Lomé, Lomé-Togo e-mail: kadambounou@yahoo.fr

**Purpose:** To design a "low-cost" tele-imaging method allowing real-time tele-ultrasound expertise, delayed tele-ultrasound diagnosis, and tele-radiology between remote peripherals hospitals and clinics (patient centers) and university hospital centers (expert center).

**Materials and methods:** A system of communication via internet (IP camera and remote access software) enabling transfer of ultrasound videos and images between two centers allows a real-time tele-radiology expertise in the presence of a junior sonographer or radiologist at the patient center. In the absence of a sonographer or radiologist at the patient center, a 3D reconstruction program allows a delayed tele-ultrasound diagnosis with images acquired by a lay operator (e.g., midwife, nurse, technician).The system was tested both with high and low bandwidth. The system can further accommodate non-ultrasound tele-radiology (conventional radiography, mammography, and computer tomography for example). The system was tested on 50 patients between CHR Tsevie in Togo (40 km from Lomé-Togo and 4500 km fromTours-France) and CHU Campus at Lomé and CHUTrousseau in Tours.

**Results:** A real-time tele-expertise was successfully performed with a delay of approximately 1.5 s with an internet bandwidth of around 1 Mbps (IP Camera) and 512 kbps (remote access software). A delayed tele-ultrasound diagnosis was also performed with satisfactory results. The transmission of radiological images from the patient center to the expert center was of adequate quality. Delayed tele-ultrasound and tele-radiology was possible even in the presence of a low-bandwidth internet connection.

**Conclusion:**This tele-imaging method, requiring nothing by readily available and inexpensive technology and equipment, offers a major opportunity for telemedicine in developing countries.

**Keywords: tele-imaging, tele-expertise, tele-diagnosis, low-cost, developing country**

## **INTRODUCTION**

Tele-radiology is the practice of radiology at a distance via the use of emerging information and communication technologies (ICTs). It consists of producing a radiological image [X-ray (XR), ultrasound (USS), magnetic imaging resonance (MRI), nuclear medicine (NM)] and making said image available to an off-site radiologist through the use of telecommunications systems for the purpose of obtaining radiological expertise that is unavailable on-site.

Dr. Kenneth Bird first conceived of tele-radiology in the late 1960s, when he used a televisual transmission system based on radio waves between the Massachusetts General Hospital (USA) and Boston's Logan airport (USA), at a distance of 5 km (1). However, Andrus was the first to formerly publish a report on the use of tele-radiology, when he successfully completed in 1972 the transmission and interpretation of images using radio waves between two sites 50 km apart (1). Since then, tele-radiology has benefited enormously from technological progress in the fields of information technology and telecommunications, and as a

result is now one of the most developed and practiced forms of tele-medicine (2–4).

The main objective of this technology is the exchange and sharing of medical images between health professionals for the purpose of obtaining a diagnosis at a distance, either in real time or following a time delay. It is perhaps one of the best practical solutions to a shortage of on-site experienced radiologists, particularly in developing countries. In the developing world in particular, the presence of radiologists is often fragmented within a country. Teleradiology therefore has the potential to play a key role in medicine in bridging the gap in equal access to diagnostic imaging.

Togo, a country with a population of six million inhabitants as of 2012, has approximately 10 radiologists, of which 90% are based at Lomé, the capital. As a result, ultrasound, a non-irradiating imaging modality, which is readily available throughout developed countries and is often used in emergency situations due to its diagnostic utility, portability, and ease of use, is rarely available in rural areas. In many developing countries, the unavailability of ultrasound is often due to both a consequence of inadequate human resources (i.e., sufficient radiologists or sonographers), as well as a lack of equipment due to, historically at least, the high costs involved in purchasing and maintaining the equipment. Recently, however, ultrasound equipment has become more affordable with the advent of low-cost Asian manufacturers (5). However, even if a radiologist or sonographer is present and available on-site, they may not be competent in all aspects of ultrasound technique. For these reasons, patients who present to rural and remote clinics with medical conditions that require radiological investigation, are often transferred to urban centers, even when their medical condition would be best managed on-site. This is often a massive burden on patients who have financial limitations and are unable to cover the costs of transport and social care.

Access to high bandwidth internet connection and the high cost of tele-radiology technology and equipment has historically represented a barrier to the use and spread of tele-radiology in developing countries, where the need is often greatest. The purpose of this study therefore was to develop an integrated tele-ultrasound, teleradiology, and tele-information method for use between experts at a tertiary medical center (CHU) and remote medical facilities (peripheral hospitals and rural clinics), of minimal cost and possible with relatively low-bandwidth internet connections, that would allow its implementation in low-income countries such as our own.

## **MATERIALS AND METHODS**

## **EQUIPMENT AND TECHNOLOGY FOR DATA TRANSMISSION IN THE PRESENCE OF HIGH BANDWIDTH: AXIS 207W NETWORK CAMERA AND AXIS 243SA VIDEO SERVER**

A network camera (Axis 207W – address IP) is installed in the remote center (patient center) and the expert center. These two cameras allow for both audio and video transmission between computers at the patient center and the expert center via an internet connection. This therefore enables videoconferencing between the two centers. An internet video server (Axis 243SA – address IP) is connected to an ultrasound machine (any commercial machine) at the patient center and allows for the real-time transmission of ultrasound video sequences via the internet to the computer at the expert center. The Axis 207W network camera et Axis 243SA video server benefit from a full package of security functions, including multiple user access levels with password protection, HTTPS encryption, IP address filtering, thus ensuring secure video handling and configuration.

## **EQUIPMENT AND TECHNOLOGY FOR DATA TRANSMISSION IN THE PRESENCE OF LOW BANDWIDTH: LogMeIn SOFTWARE**

LogMeIn is a remote access software. It enables one to connect to a host computer from another computer or device (client) at any time, as long as an internet connection is available. Two LogMeIn products exist: LogMeIn Free and LogMeIn Pro. LogMeIn Free is free online and LogMeIn Pro requires a subscription. Following installation of the LogMeIn Pro software on the host computer, it is possible to access the software from any internet enabled computer or mobile device (LogMeIn for iOS or LogMeIn Ignition for Android).

LogMeIn Pro has a multitude of functions, including transmission of high definition quality video, audio transmission, file transfer, file and desktop sharing, control of multiple monitors, password saving, alerts with LogMeIn central, remote awakening from standby, and system diagnostics.

The computers from which one is running and remotely accessing LogMeIn must meet the following system requirement:

## **Host system requirement**


## **Client system requirements**


The LogMeIn software, by allowing a radiologist at the expert center to take control of the computer at the patient center, via laptop for example, renders possible not only still image transfer (XR, USS, CT, MRI, NM), but also tele-ultrasound expertise in the presence of an inexperienced sonographer or untrained user at the patient center. In the absence of a sonographer at the patient center, an expert is able to make a radiological diagnosis in delayed time from ultrasound video sequences sent by an untrained user, with the use of the 3D virtual navigation program ECHO-CNES (Unité de Médecine et Physiologie Spatiales de Tours) (6). For tele-ultrasound, the computer at the patient center (isolated site) is beforehand connected to the ultrasound device via a video converter/USB (Pinnacle for example). Almost all the ultrasound devices have a port (bearing) for video converters.

LogMeIn includes a security system not only based on a 256 bit SSL/TLS encryption, but also multiple authentication systems. In effect, LogMeIn has many systems of authentication, including authentication of the gateway to the Client, authentication of users to the gateway, authentication of the gateway to the host, and authentication of the host to the gateway, ensuring the security of the computer data at both the patient and expert center.

Finally, LogMeIn can be coupled with Skype to enable audio and face-to-face communication between the operators at the two centers. Skype is a free software available online that includes a call notification system and contacts lists.

## **PILOT EXPERIMENT FOR THE TELE-RADIOLOGY PLATFORM**

For this pilot study, the remote patient center was CHR Tsévié situated 35 km from Lomé, capital of Togo. CHR Tsévié has a general medicine service, an internal medicine service, and a radiology service that includes a 2D ultrasound machine (GE Logiq 200) but no on-site radiologist. The expert centers were CHU at Lomé and CHU Trousseau de Tours in France (approximately 4500 km from Tsévié). An internet connection was installed at CHR Tsévie (fiberoptic) and at CHU Lomé (ADSL) for the study. The transmission speed of the internet connection had been measured by online free software such as Speed Test. The approximate averages of the frame rate and delay transmission of the ultrasound video sequences determined when the actual speed measured is very close to that theoretical claimed by the Internet Service Provider were considered. The CHU at Lomé is the second biggest tertiary referral center in Togo. The system was tested with 50 patients at CHR Tsévie. Patients gave full informed consent. These patients were either recruited upon emergency admission to hospital or were already hospitalized at CHR Tsévie. Diagnostic imaging requests were ordered initially by the doctors at CHR Tsévie or following a telemedicine consultation. The experts were university hospital radiologists. The imaging requests were ordered predominantly by general medicine doctors at CHR Tsévie.

The quality of the images tele-transmitted were appreciated by three expert radiologists (University hospital radiologist), the appreciation retained for the quality of the transmitted images for every bandwidth was that of at least two of the three expert radiologists.

## **RESULTS**

A bandwidth of a minimum of 1 Mbps was necessary for the transmission of real-time ultrasound video sequences, but also background video from the remote center with the Axis technology. The quality of the ultrasound images tele-transmitted by the video server Axis 243SA was sufficient (minimum frame rate about 10 fps) for an accurate diagnosis with a transmission delay of approximately 1.5 s. The optimal quality of transmission (approximate average frame rate of 35 fps) was obtained with a bandwidth of 4 Mbps, which transmission delay was about 0.5 s.

With an average bandwidth of 512 kbps, LogMeIn allowed us to transmit images and video sequences of satisfactory quality with a delay of 2 s. With a bandwidth of 256 kbps, we noticed with Log-MeIn a distortion of color ultrasound video sequences, although static images were of acceptable quality. LogMeIn was also tested with 3G dongle internet connections (SFR, Helim de TogoTelecom). Video sequences and static images were of satisfactory quality using this.

The LogMeIn file transfer function permitted the direct transfer of video files saved at the patient center computer to the computer at the expert site, enabling delayed post-treatment with the ECHO-CNES navigation program. A file of 100 Mb in size took about 3 min to transfer.

The use of Axis technology with an average internet connection of 2 Mbps enabled the experts at the CHU campus and the CHU Trousseau de Tours en France to perform highly satisfactory realtime tele-ultrasound consultations for 28 ultrasounds: abdominal (*n* = 10), pelvic (*n* = 6), obstetric (*n* = 4), prostate (*n* = 4), and breast (*n* = 4) (**Figure 1**). The coupling of LogMeIn with Skype with a bandwidth of 512 kbps allowed for both tele-ultrasound and videoconferencing capabilities in 15 cases (**Figure 2**). For these tele-ultrasound videoconferences, the ultrasound operator at the CHR Tsévie was either a radiology intern sent to the CHU specifically for the case or the gynecologist at CHR Tsévie. We

were also able to perform seven cases of tele-radiology using Log-MeIn relating to conventional XR, computer tomography, and mammograms (**Figure 3**).

With inexperienced ultrasound operators at CHR Tsévie (e.g., radio operators, nurses, midwives), 10 delayed-time diagnostic tele-ultrasound cases were performed with the virtual navigation program ECHO-CNES. These tele-ultrasound sessions enabled a degree of gradual training at a distance of non-experts (e.g., midwives).

## **DISCUSSION**

The Axis 207W camera server and the Axis 243SA video internet server are readily commercially available. The Axis 207W camera server was originally designed and intended for indoor video surveillance and to be able to be controlled remotely. The Axis 243SA video internet server is used in any facility where an analog surveillance system is already installed but the passage in the digital technology is imperative. This is the first time Axis video internet servers and technology have been used in medical tele-radiology and represents an innovative use.

The Axis 243 video server costs only 850 euros and the Axis 207 video camera costs only 350 euros. 1500 Euros is therefore enough using this technology to implement a tele-radiology platform between a remote center and a center of expertise. The monthly cost of internet connection in Togo actually for 512 kbps is about 46 euros, for 1 Mbps is 89 euros, and for 2 Mbps is 172 euros. These costs, which to date are one of the most expensive in West Africans countries, will certainly be reduced in the next coming years in the wake of the new policy of telecommunication committed by the government. We have previously used Axis technology in conjunction with a cloud based file hosting service (Dropbox) as a low-cost platform for obstetric and gynecological tele-imaging (7). The preliminary technical and clinical results have also been published (8).

The Axis technological infrastructure requires a minimum bandwidth of 1 Mbps for a transmission of acceptable quality. While this level of bandwidth is relatively small when compared with the real-time cardiological tele-consultation and teleechocardiography platform proposed by Boman, which requires a bandwidth of 20 Mbps (9), such a bandwidth will not necessarily be available in rural zones in developing countries (10). One of our aims therefore was to conceive of a system based around the Log-MeIn software, which is not only affordable, but also functional in areas with low-bandwidth internet, such as developing countries like Togo.

LogMeIn is available online and the basic version is free. The pro-version costs only 53 euros per year per computer. Other than the low cost, the high level of security available with LogMeIn compared with many other remote access programs was another factor that attracted us to this piece of software. LogMeIn has two intrusion detection capabilities: SSL/TLS et the LogMeIn intrusion filters. The first level of intrusion detection is based around SSL/TLS in order to detect the possible modification of data in transit. The second layer of security relies on three intrusion filters, namely an IP address filter, a denial of service filter, and an authentication filter.

The file transfer capability available with the LogMeIn Proversion, which allowed transfer of ultrasound files up to 100 Mb in around 3 min, demonstrates clearly that this software can be used for delayed tele-ultrasound diagnosis with ECHO-CNES in the presence of a non-expert at the patient center.

The LogMeIn technological infrastructure, which functioned perfectly well with 3G dongles and modest bandwidth offers therefore a practical mechanism for medical tele-imaging services even in the most remote areas in which information and communications technologies are lacking. Further, tele-ultrasound expertise could be made available for patients in ambulances with the use of the LogMeIn software. All that is required is a portable ultrasound on board the ambulance and a 3G dongle for internet connection. One can also envisage the expert being able to view the images on a tablet device or smartphone with the use of LogMeIn for iOS or LogMeIn for android.

With untrained ultrasound operators at CHR Tsévié, only delayed tele-ultrasound diagnosis with the virtual navigation program ECHO-CNES was possible. This 3D reconstruction program, of which the diagnostic accuracy for general abdomen ultrasound was estimated at 91% in a previous study (6), enables the expert to review the images sent by the untrained person at a later time and make a radiological diagnosis. Ultrasonography involves a degree of technical skill and precision, and thus it is quite operator dependent. As a result, it can be difficult to guide a lay person remotely by voice and/or video. The difficulty in realizing real-time tele-ultrasound with an untrained operator has led some authors to produce visual guides to assist in obtaining images of the organ of interest for diagnosis (11, 12). Sheehan et al., for example, have produced a guide (Expert Visual Guidance) (11) consisting of a program that shows the various positions and angles of the ultrasound probe necessary to obtain the required views of the organ of interest. The authors tested the program with 20 medical students with no prior ultrasound experience. Compared to a group of medical students who were guided by voice alone, these untrained students were more competent at obtaining the necessary views for making a diagnosis.

Other authors have used robotic systems, ESTELE (13), OTELO (14),TER (15),to allow an expert to remotely manipulate the ultrasound probe on the patient. These remote-control systems include a robotic arm at the end of which is fixed an ultrasound probe, which an untrained person places and maintains on the patient. The radiologist at the expert center holds a fictitious probe, and via this it is able to tele-manipulate the ultrasound and obtain different ultrasound views in real time. The robotic arm at the patient center reproduces with high fidelity the hand movements of the radiologist at the expert center. The high cost of such robotic systems, however, renders them unaffordable in low-income countries such as Togo.

The CHU Trousseau de Tours in France was one of the expert centers in this study. The ability to have specialists make a diagnosis at a distance from France thanks to our method would significantly improve the care of certain patients and avoid the need for costly repatriations to European centers in some cases. The beneficial role that such a platform can play was apparent when the "télésanté en Afrique" project was piloted by Rovetta et al. in 1997, in which doctors in France and Italy performed successful tele-medicine consults with hospitals in the Ivory coast and South Africa (16). Further, Gimel et al. (17) demonstrated the feasibility of a tele-pathology service between dermo-pathologists at the Massachusetts General Hospital in the United States and African hospitals, via the transmission of static histology images. The availability and combination of free, yet secure, software such as LogMeIn and low-cost equipment such as Axis technology and their use in developing countries, as used in this study, will hopefully further stimulate the introduction of tele-medicine networks between reference centers in developed countries and remote centers in developing countries.

The fact that both patient and medical personnel adherence was good in this study is highly encouraging. Several other telemedicine projects used throughout the world, although innovative, have ultimately failed due to a lack of interest by users (18).

We have not as yet carried out a study in which we had sufficient *n* numbers to achieve statistical significance with Cohen's Kappa (a limitation of this current study). Nevertheless, our work appears reproducible. However, we organized a workshop during the ninth Société de Radiologie de l'Afrique Noire Francophone (SRANF) in May 2011 in Lomé in which we successfully demonstrated, in particular, tele-ultrasound using our system. Numerous eminent African radiologists carried out tele-ultrasound sessions from the congress hotel with patients at remote centers and were satisfied that the quality of the ultrasound images was sufficient for an accurate diagnosis.

The preliminary results from this study suggest that the functionality of our low-cost tele-radiology method is satisfactory. Nonetheless, future studies using larger sample sizes will be required for clinical validation and to determine the feasibility of our method on larger scales.

## **CONCLUSION**

In this pilot study, we configured a system for medical tele-imaging using hardware that was low cost and online software that was free, demonstrating its feasibility in low-income countries. The results were highly satisfactory and offer encouraging potential for the development of telemedicine in developing countries. Such countries often suffer from a chronic shortage of medical specialists and their means are often limited in comparison to their huge healthcare needs. Despite its low cost and ease of use, the political will needs to exist to enable the large scale and systematic roll out of a method such as ours in remote areas in developing African countries.

## **ACKNOWLEDGMENTS**

We thank the Agence Universitaire de la Francophonie (AUF).

## **REFERENCES**

1. Andrus WS, Bird T. Teleradiology: evolution through bias to reality.*Chest* (1972) **62**(6):655–7. doi:10.1378/chest.62.6.655


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 12 June 2014; accepted: 20 August 2014; published online: 05 September 2014. Citation: Adambounou K, Adjenou V, Salam AP, Farin F, N'Dakena KG, Gbeassor M and Arbeille P (2014) A low-cost tele-imaging platform for developing countries. Front. Public Health 2:135. doi: 10.3389/fpubh.2014.00135*

*This article was submitted to Public Health Education and Promotion, a section of the journal Frontiers in Public Health.*

*Copyright © 2014 Adambounou, Adjenou, Salam, Farin, N'Dakena, Gbeassor and Arbeille. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

## The Pacific Island Health Care Project

## **Donald Ames Person\***

Tripler Army Medical Center, Honolulu, HI, USA

## **Edited by:**

Richard Wootton, University Hospital of North Norway, Norway

#### **Reviewed by:**

Donghua Tao, Saint Louis University, USA Laurent Bonnardot, Paris Descartes University, France

**\*Correspondence:** Donald Ames Person e-mail: dperson001@att.net

**Introduction/Background:** US Associated/Affiliated Pacific Islands (USAPI) include three freely associated states: Marshall Islands, Federated States of Micronesia, Palau, and three Territories: American Samoa, Guam, and Commonwealth of the Northern Mariana Islands.

**Objective:** The Pacific Island Health Care Project (PIHCP) provides humanitarian medical referral/consultation/care to >500,000 indigenous people of these remote islands.

**Methods:** In the mid-1990s, we developed a simple store-and-forward program to link the USAPI with Tripler Army Medical Center. This application allowed image attachment to email consultations.

**Results:** More than 8000 Pacific Islanders have benefited from the program. Three thousand Pacific Islanders prior to telemedicine (1990–1997) and since store-and-forward telemedicine (1997-present), the PIHCP has helped an additional 5000. Records post dynamically and are stored in an archival database.

**Conclusion:** The PIHCP is the longest running telemedicine program in the world delivering humanitarian medical care. It has bridged the Developing World of the remote Pacific Islands with advanced medical and surgical care available at a major US military teaching hospital. (The opinions expressed here are those of the author and not that of the Army, Department of Defense, or the US Government.)

**Keywords: telemedicine, teleconsultation, tele-referral, Pacific Island, Pacific Islands Healthcare Project**

## **INTRODUCTION/BACKGROUND**

The former US Pacific Trust Territories include Republic of the Marshall Islands, Federated States of Micronesia (FSM) (Chuuk, Kosrae, Pohnpei, and Yap States), and Republic of Palau. American Samoa in the South Pacific, Guam, and Commonwealth of the Northern Mariana Islands (CNMI) make up the remainder of the US Associated/Affiliated Pacific Islands (USAPI). American Samoa, CNMI, and Guam are US Territories. These remote nations, home to about 500,000 indigenous persons, include over 2200 islands and atolls scattered across more than 7 million square miles of Pacific Ocean (Figure S1 in Supplementary Material). These countries are part of the developing world of the Pacific, several lacking adequate infrastructure, transportation, communications, and medical support services. The people live in few of the islands and atolls of Micronesia in small family units. Many survive as subsistence fishers and farmers. Historically traditional healers, village elders, a few expatriate physicians, and nurses provided medical care to the people. An Institute of Medicine study detailed problems of medical care and referral from the US Pacific (1). The senior leadership (commander, department chairs, teaching chiefs, medical education directors) at Tripler Army Medical Center (TAMC) located in Honolulu, Hawaii had long recognized the medical need and unique teaching value of Pacific Islanders to our mission to provide for graduate medical education (GME). With training programs in surgery, medicine, pediatrics, psychiatry, otolaryngology (ENT), urology, orthopedics, and pathology it was incumbent on the leaders to find a way

to organize, fund, and regulate the flow of patient cases from the Pacific. With the granting of full sovereignty to the jurisdictions in 1986, there was an influx of sick patients sent to TAMC from the islands. The Compact of Free Association provided for support of health and welfare in the jurisdictions and the provision of limited medical care at TAMC, as available, for selected Pacific Islanders on a reimbursable basis. The jurisdictions of the US Pacific are very poor and even partial reimbursement was unreasonable in the 1980s, much less now! In 1988, several of us began to discuss possible solutions to provide much-needed humanitarian care to Pacific Islanders at TAMC and at the same time enhance GME. Federal medical facilities neither make a profit nor provide charity. We presented our proposal to the late US Senator Daniel K. Inouye, an ardent supporter of Army medicine and a champion of the underserved peoples of the US Pacific, and he provided the means. Under his aegis, the US Congress appropriated funds for the travel of a limited number of Pacific Islanders (100 patients annually) to/from TAMC for medical care. TAMC bore all medical costs associated with the program. The Pacific Island Health Care Project (PIHCP) accepted its first patients in mid-1990. Almost since its inception, I have directed the program. Major referral sites included Majuro Hospital, state hospitals in Pohnpei, Chuuk, Kosrae, Yap, and Belau National Hospital, Palau. Physicians in the islands made referrals by mail, telephone, and facsimiles. Distance, an International Date Line, five time zones, unreliable communications, and language differences posed substantial problems. The tremendous cost of caring for hundreds of

sick Pacific Islanders (300–450/year) threatened continuation of the program. In-patient costs of \$50,000–\$100,000/patient were commonplace. Difficulties with communication, misunderstanding of program limitations, etc., added to the confusion. Patients with disseminated malignancies, multiple congenital anomalies, heart failure, active tuberculosis, end-stage renal disease, stroke, etc., arrived at TAMC unannounced. Some patients died en route, others arrived *in extremis*. Despite these problems, nearly 3,000 Pacific Islanders were cared for under the PIHCP in the early years of the program and prior to telemedicine.

## **OBJECTIVES**

The PIHCP was developed to (1) provide humanitarian medical care to deserving, underserved, indigenous peoples of the USAPI at a reasonable cost, restore them to health, and return them to their island homes as productive members of society and (2) select patients to enhance GME for TAMC's residents-in-training and staff (i.e., good teaching cases).

## **DESIGN/METHODS**

Real-time (synchronous) medical consultations made between Kwajalein Headquarters (US Army Missile Defense Command located in the Marshall Islands) and TAMC as early as 1992 using the AT&T Video teleconferencing system validated the ease of consultation (2). The utility of the Picasso telephone, demonstrated by me at the inaugural meeting of the Pacific Basin Medical Association (PBMA) in Pohnpei in 1995, confirmed that storeand-forward (asynchronous) consultation was just as easy to use (3) (Figure S2 in Supplementary Material). Excessive cost and needed technical expertise precluded the sustainability of either of those systems in the Pacific. Electronic mail with attached images provided a suitable medium for patient consultation and referral. With the technical assistance of Project Akamai, I was able to develop a simple store-and-forward method of telemedical consultation/referral. During the fourth Annual Meeting of the PBMA in Weno, Chuuk (1998), we demonstrated the utility of the Internetbased consultation/referral platform. Early results from four test sites confirmed the ease and utility of the web-based system (4, 5). TAMC's judge advocate (senior attorney) and I discussed legal ramifications of PIHCP telemedicine program many times and decided that a consent statement/form be added to each patient's consultation page. Indigenous island people do not speak English, so it was incumbent on the referring provider to assure patient understanding and consent.

## **RESULTS**

We established 10 workstations in the USAPI between February 1998 and January 2001 (Table S1 in Supplementary Material). Fifty percent of referred cases (5422) came from Palau and the Marshalls (Table S2 in Supplementary Material). Males and females are equally represented, a quarter <18 years, a third >50 years, and the remainder, 18–50 years of age (Table S3 in Supplementary Material). Table S4 in Supplementary Material lists the departmental referrals. The majority of patients were referred to the Department of Surgery (general/oncologic, pediatric, vascular, plastics/reconstructive, neurosurgery, cardiothoracic, etc.). Subspecialty medicine referrals were next most frequent followed

by "other" that includes ENT, ophthalmology, urology, radiology, dental, psychiatry, and pathology. Pediatric referrals were next. Gynecological malignancies made up the bulk of the cases referred to the Department of Obstetrics/Gynecology. Historically, 792 users had access to the website. Currently, there are 158 active users: American Samoa (5) Chuuk (5), CNMI (5), Ebeye (7), Guam (4), Kosrae (2), Majuro (14), Palau (17), Pohnpei (7), Yap (7), and 85 from the US. Except for a few nurses and administrators, all TAMC users are specialists or subspecialists. Island referring physicians are mostly generalists, some with additional certifications in surgery, medicine, pediatrics, or obstetrics/gynecology. A statistical module posts all data dynamically. All cases (patient web pages) with attached imagery (photographs, EKGs, x-rays, CT and MRI scans, laboratory results, histopathologic reports, echocardiograms, full motion video, etc.) are archived and retrievable (6, 7). Currently, about 20 cases are submitted monthly, during the early years, 30/month. The database is searchable using key words, terms, diagnoses, and phrases.

## **VIGNETTES FROM THE PIHCP**

Diseases, once common in developed countries 50–100 years ago, but now mostly eradicated, are still prevalent in Oceania. Infectious diseases, such as tuberculosis, leprosy, rheumatic fever, leptospirosis, dengue, tropical pyomyositis, and necrotizing fasciitis, are common. Some unique conditions relate to culture and environment, such as billfish injuries, betel nut cancers, marine envenomations, motor boat injuries, burns, electrocution, are examples. Pacific Islanders also suffer from diseases associated with modern living; hypertension, diabetes, hyperlipidemia, gout, obesity, coronary artery disease, and cancer. Store-and-forward telemedicine is uniquely suited to consultation and referral from remote, resource poor areas of the developing world. In the early days of the program, few of the jurisdictions had more than antiquated xray machines. Film/developer was (and still is) often unavailable. There were no obstetrical ultra sound machines, much less CT scanners! Currently, three jurisdictions have CT scanners (Palau, Majuro, and American Samoa). Guam and Saipan (CNMI) have advanced and sophisticated imaging capabilities including CT and, MRI.

## **OTOLARYNGOLOGY/HEAD AND NECK SURGERY (ENT)**

Common ENT referrals from the Pacific Islands include children with cleft lip/palate (Figure S3 in Supplementary Material). Oral cancers are especially prevalent in Palau, Yap, and Pohnpei, Islands where both men and women chew betel nut mixed with tobacco and slaked lime. Surgical management of these conditions provides ENT and maxillofacial residents with the skills necessary to treat injuries common in the war zone. A young boy from Kosrae fell on a coconut husker (spike/adze buried in the ground) and lacerated his trachea. Total body emphysema ensued and threatened his life. Our pediatric ENT surgeon performed a tracheostomy that allowed the air to dissipate. The child returned home a week later, cured (Figure S4 in Supplementary Material).

## **UROLOGY**

Renal stones (nephrolithiasis), common in Pacific Islanders are easily treated with lithotripsy, unavailable in the islands. Large staghorn calculi, however, require surgical removal and provide excellent resident training. Rare male genital malignancies provide staff and residents with unique learning opportunities. TAMC's urogynecologist collaborates with reconstructive urologists on complicated female cases. Hypospadias, epispadias, exstrophy of the bladder, cryptorchidism, and ambiguous genitalia challenge surgeons, as well as pediatric endocrinologists. We identified a Yapese family several years ago where three children with ambiguous genitalia had congenital adrenal hyperplasia. We reported a child from Chuuk with ambiguous genitalia that developed two malignancies (Li-Fraumeni syndrome) (8). A Chuukese boy fell and lacerated his perineal urethra. The photographic images attached to his case allowed our pediatric urologist to recommend temporizing measure (suprapubic cystostomy/wound care) prior to definitive repair (Figure S5 in Supplementary Material). Based on supporting images urology consultants are able to determine which cases to treat robotically.

## **ORTHOPEDICS**

Traumatic injuries, chronic fractures and dislocations, developmental anomalies, chronic osteomyelitis, musculoskeletal tumors, tropical pyomyositis, arthritis, spondylitis (including Potts disease), disk disease, and spinal stenosis are common in Pacific Islanders. Musculoskeletal cancers, uncommon in the US are frequent in Pacific Islanders. The treatment of patients with osteosarcoma, chondrosarcoma, Ewing sarcoma, rhabdomyosarcoma (Figure S6 in Supplementary Material), giant-cell tumors, aneurysmal bone cysts, and other rare musculoskeletal tumors enhance resident training. Photos, x-rays, ultra sound, and CT images enhance consultations. A chief orthopedics resident published our experience, highlighting the educational benefits to the orthopedic program of the PIHCP (9). These patients provide a wealth of experience for orthopedic residents and many of the skills learned are directly applicable to combat requirements.

## **CARDIOTHORACIC SURGERY**

Acute rheumatic fever and rheumatic heart disease (RHD) occur frequently in children and young adults from the islands. Recurrent group A, β-hemolytic streptococcal infections lead to the development of valvular heart disease. Utilizing telemedicine, we can select those patients who would likely benefit from valve replacement surgery (compliance with penicillin prophylaxis). Most islanders have bioprosthetic valves placed. A pediatric resident and I recently reviewed and published our experience with RHD in Pacific Islanders (10). These consultations are enhanced with photographs (cyanosis, clubbing, anterior chest bulge) chest x-rays, EKGs, and ECHO video clips.

## **NEUROSURGERY**

Neurosurgical consultation is helpful and lifesaving in selected cases. Patients with myelomeningocele, hydrocephalus, paralysis, stroke, chronic and degenerative nervous system disease, etc., are not accepted. Long-term care and rehabilitation is not available. Our neurosurgeons have used endoscopic third ventriculostomy in selected cases and radiosurgery (gamma knife) in a few others. Treating patients with lesions amenable to neurosurgical intervention can be challenging but rewarding (11).

Photographs, x-rays, CT scans, head ultrasound images, and full motion video clips attached to the consultation can be diagnostic.

**GENERAL, ONCOLOGIC, PEDIATRIC, PLASTIC, AND VASCULAR SURGERY** Our surgical training programs probably benefit the most from referrals from the USAPI. Several years ago, we consulted on a young Marshallese woman with recurrent hypoglycemia and a breast tumor. Her cystosarcoma phyllodes tumor secreted an insulin-like factor and mastectomy resulted in normalization of her blood sugar (12) (Figure S7 in Supplementary Material). In another remarkable case, surgical oncologists and orthopedic surgeons removed a massive osteosarcoma from the right shoulder a woman from Kapingamarangi (outer island in Pohnpei State) [Ref. (13), Figure S8 in Supplementary Material]. Pediatric surgeons treat children from the Pacific with diaphragmatic hernias, Hirschsprung's disease, tumors [Ref. (14, 15), Figures S9 and S10 in Supplementary Material], congenital anomalies, and acquired conditions. Minimally invasive surgical techniques employed in recent cases allows for rapid recovery and short hospitalizations.

## **OBSTETRICS AND GYNECOLOGY**

Telemedicine is especially useful in assessing patients with gynecological malignancies. Women with advanced, but treatable uterine and ovarian cancers provide a wealth of experience for our OB/Gyn training program (16). Women with gestational trophoblastic malignancies including choriocarcinoma, hydatidiform mole, and molar pregnancies have also benefited from the PIHCP. In some cases, consultant advice is all that is necessary. A pregnant Chuukese woman had a 0.22 caliber bullet lodged in her frontal lobe. Based on the teleconsultation, both neurosurgeon and perinatologist agreed that operative removal was contraindicated (Figure S11 in Supplementary Material). Sadly, shortly before delivery at home she suffered a grand mal seizure, delivered a stillborn, and died. Another pregnant woman with a massive umbilical hernia and peritoneal tuberculosis (tabes mesentarica) was consulted because of concerns for herniation and torsion of her gravid uterus. Photographs of her abdomen provided TAMC consultants (general surgeon and perinatologist) the information needed to assure the remote provider that it was safe to treat her tuberculosis and let her continue with her pregnancy. In another case, a 38-year-old Marshallese woman referred to TAMC had a 90 pound ovarian serous cystadenoma removed by our gynecological surgeons (Figure S12 in Supplementary Material).

## **DISCUSSION/CONCLUSION**

Internal medicine and pediatric consultants are often the first to evaluate patients referred to the PIHCP. Many adult patients have significant comorbidities and successful surgical treatment depends on careful selection and pre-operative, perioperative control of factors such as hypertension, diabetes, renal insufficiency, hyperuricemia, hypoalbuminemia, hypokalemia, and other electrolyte disturbances, prior to and during surgery. Pre-operative dental extractions/restorations have contributed to the long-term success of our open-heart program. Infectious diseases, such as tuberculosis, leptospirosis, and leprosy are still prevalent in the Pacific Basin and prior to telemedicine; children with various

forms of extra pulmonary tuberculosis (arthritis, osteomyelitis, spondylitis, mastoiditis, meningitis, etc.) came to TAMC for treatment. Such patients flew into Honolulu and presented in the TAMC emergency department causing disruption, delays, and the institution of emergency isolation procedures. Tuberculin skin tests/chest x-rays have been required for air travel from the Pacific islands to Hawaii for many years (PIHCP Referral Form). Since telemedicine, no patients with active TB are accepted. Now remote teleconsultation suffices for the diagnosis and treatment of Pacific Islanders with tuberculosis. The referral of an 18-year-old nursing student from Ebeye who was dying of a wasting disease is a case in point. We diagnosed adrenal tuberculosis (Addison's disease) based on her exam, photographs, x-rays, labs, and ultra sound images. She was treated and restored to health (17). The uniqueness of the PIHCP allows for a better understanding of the epidemiology of both infectious and noninfectious diseases. Leptospirosis, prevalent in many of the islands of the Pacific reaches almost epidemic proportions in Kosrae State. Prior to the development of telemedicine, we summarized our experience with the diagnosis and treatment of acute leptospirosis with renal failure in children from the islands who were treated in Hawaii (18). Based on that experience and utilizing telemedicine, we are now able to remotely diagnose leptospirosis, recommend penicillin, fluid restriction, and supportive care. Children with leptospirosis no longer require medical evacuation. Similarly, children with tropical pyomyositis were sent urgently to TAMC, without approval, some unconscious and *in extremis*. Tropical pyomyositis, an acute pyogenic infection of muscle caused by *Staphylococcus aureus*, untreated, spreads, sepsis ensues, and many patients die. Since the introduction of telemedicine, island surgeons now know how to treat the condition aggressively with incision/drainage and anti-staphylococcal antibiotics. Teleconsultation allows these children to be treatment at home. Another great success of telemedicine is the surgical treatment of patients with perineal necrotizing fasciitis (Fournier's gangrene). Using interactive PIHCP telemedicine, a surgical colleague in Majuro carefully documented (with intraoperative photographs) the extent and involvement of his patient's infection. TAMC urology, surgery, and plastics consultants weighed in and recommended wide excision, diverting colostomy, burial of the testicle deep in his groin, and ultimately skin grafting. The surgeon presented a second patient and after a similar interaction and successful outcome, now Fournier's gangrene cases are managed locally. Several years ago, two little girls fractured their femurs, one hit by a car (Majuro), the other fell out of a second story window (Pohnpei). Surgeons at both locations did what they could to provide traction without special equipment or orthopedic support. Each submitted his case to the PIHCP website (attached x-rays, photos, etc.) I was able to facilitate their consultations with TAMC's pediatric orthopedist and we successfully managed both cases remotely at a great savings both emotionally and monetarily (19).

Some of my concerns for the future of the PIHCP include burgeoning telecommunications technology, advances in medical equipment/device development, drug/medication development, changes in medical care delivery, and changes in physician education. Complex military cyber security requirements threaten

access to the PIHCP by distant providers. Hospitals in remote areas cannot afford or do not have access to advanced imaging technologies. Hospital laboratories are inadequate. Basic drugs, medications, and other pharmaceuticals are in short supply, unavailable, or unaffordable. Radiologists, pathologists, pharmacists, etc., are lacking. The delivery of health care in the US has evolved from hospital-based to clinic-based/day surgery outpatient care. Transvascular, endoscopic, and minimally invasive techniques have markedly changed modern health care practice in the US. Pace makers, implantable devices, shunts, pumps, etc., require specialized care and are contraindicated for use in Pacific Islanders, as follow up care is not guaranteed. On a more optimistic note, the US military has long been involved in disaster relief, civil affairs, medical civic action, and nation building. The PIHCP offers invaluable training, experience, and a better understanding of the people and conditions in the Developing World to students, residents, and staff prior to inevitable deployments to remote areas around the globe where they will be engaged in medical care similar to that provided patients from the USAPI. Cases from the PIHCP often challenge our basic observational, interpretive, integrative, and diagnostic skills. Caring for grateful Pacific Islanders with complex, advanced, medical conditions, restoring them to health, and returning them home as productive members of society is a truly rewarding experience. PIHCP telemedicine has bridged the gap between the distant Pacific and a major medical center.

## **SUPPLEMENTARY MATERIAL**

The Supplementary Material for this article can be found online at http://www.frontiersin.org/Journal/10.3389/fpubh.2014. 00175/abstract

## **REFERENCES**


to orthopedic graduate medical education at Tripler Army Medical Center. *Mil Med* (2005) **170**:214–8.


disease in Ebeye, Republic of the Marshall Islands. *Hawaii Med J* (2009) **68**(1): 8–11.


**Conflict of Interest Statement:** The author declares that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 18 July 2014; accepted: 19 September 2014; published online: 13 October 2014.*

*Citation: Person DA (2014) The Pacific Island Health Care Project. Front. Public Health 2:175. doi: 10.3389/fpubh.2014.00175*

*This article was submitted to Public Health Education and Promotion, a section of the journal Frontiers in Public Health.*

*Copyright © 2014 Person. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

## Pediatric teleradiology in low-income settings and the areas for future research in teleradiology

## **Savvas Andronikou\***

Department of Radiology, Faculty of Health Sciences, University of the Witwatersrand, Johannesburg, South Africa

## **Edited by:**

Richard Wootton, University Hospital of North Norway, Norway

#### **Reviewed by:**

Saskia Spijker, Epworth Medical Imaging, Australia Kokou Toussaint Fofo Adambounou, University of Lomé, Togo

#### **\*Correspondence:**

Savvas Andronikou, Medical School, York Road, Parktown, Johannesburg, Gauteng 2193, South Africa e-mail: docsav@mweb.co.za

Teleradiology is an established mechanism to overcome the lack of on-site radiologists and can benefit children in developing countries. In this "perspective" on teleradiology for pediatric care in underdeveloped countries, three low-cost teleradiology programs are discussed from experiences of one teleradiologist, in relation to previous publications on this subject. Key issues discussed include mechanisms for sustainability, cost-effectiveness, resources, and barriers to success. Reliance on each link of a telereading chain is highlighted as a constant source for concern.

**Keywords: teleradiology, tuberculosis, pulmonary, HIV infections, developing countries, resource allocation, cost-effectiveness, X-rays**

## **INTRODUCTION**

In low-income settings, teleradiology has a significant role to play in the diagnosis and management of respiratory diseases in children. Teleradiology has been shown to improve the diagnosis of tuberculosis, especially in settings with a high burden of HIV infection (1).

Limited radiology services are a major obstacle to health care for sick children. Some sub-Saharan African countries have no radiologists in public service (1). This is when teleradiologic interpretation of diagnostic imaging can circumvent the need for on-site radiologists; children in underserved areas benefit from X-ray reports by expert radiologists, without having to travel and with quick results offering the opportunity for earlier and more appropriate treatment (1–3).

For a telereading program to be successful, it must be costeffective for an underserved population, overcome technical, legal, and language barriers, and lead to improved outcomes sustainably. This paper describes my personal experience in three different teleradiology programs. I will highlight possibilities, imaging quality, and major obstacles to providing a sustainable service from the point of view of a pediatric radiologist, and make research recommendations that explore the expanding role of teleradiology for children in low-income settings.

## **THREE DIFFERENT EXPERIENCES WITH TELERADIOLOGY TELEREADING FOR MÉDECINS SANS FRONTIÈRES**

## **Intersectional radiologist and ad hoc MSF telereader**

In 2009, I was employed by Médecins Sans Frontières (MSF) as "intersectional radiologist." In addition to administrative tasks, I was mandated to promote the use of diagnostic imaging consultation through e-mail across all operational centers.

I spent a significant portion of my time explaining that the lack of expertize in the interpretation of X-ray and ultrasound could be overcome through digitizing the images and referring them to an expert radiologist by e-mail. The process of digitizing hard copy film from existing X-ray units with wet developing was the first obstacle to overcome. Referrals gave me concerns both with regard to the expectations of the clinicians and the quality of the imaging (**Figure 1A**). A colleague receiving regular referrals from a pediatric MSF site using a digital unit in Liberia had more success and satisfaction, managing to assist the clinicians regularly with management changing diagnoses.

I re-oriented my efforts into educating control centers on the benefits of digital imaging; promoting CR conversions of existing units; establishing digitization guidelines for hard copy films; and improving imaging quality through site visits. Subsequent to my departure, two formal teleradiology pilot programs were set up.

## **Formal MSF telereading pilot projects**

Médecins Sans Frontières teleradiology has evolved from an *ad hoc*, e-mail based X-ray consultation service, into a structured platform-based service. I have been pediatric teleradiologist for MSF using the Collegium Telemedicus platform (4), since its creation. Other telereaders also volunteer, at no cost to MSF, reporting X-rays from number of MSF field projects. Most MSF equipment is supplied by the local Ministry of Health, using hard copy film and wet developing. To enable telereading, hard copies were photographed against viewing boxes using a digital camera. JPEG images were transmitted via Collegium Telemedicus for reporting. More recent MSF X-ray equipment in Sub-Saharan Africa is digital and can be transmitted unchanged for telereading.

In 2012, 818 plain X-ray images from 14 sites were sent for teleradiology opinion. A significant proportion of these (36%) were pediatric cases. I have provided opinions on 72 pediatric imaging referrals from 24 July 2012 to April 2014 from Malawi, Central African Republic, Uganda, Guinea, Tajikistan, and Cambodia. These were mainly chest radiographs from HIV projects for diagnosing tuberculosis. The quality of radiographs varied from very poor (**Figure 1B**) to diagnostic (**Figure 1C**), yet I was able to assist the clinicians and engage in dialog with them, which was satisfying. I was also able to assist in the development of the diagnostic imaging manual for the organization.

**FIGURE 1 | A variety of pediatric telereading referrals**. **(A)** An X-ray referral from an MSF site in Tajikistan was received as a JPEG created by holding the hard copy X-ray up against a window and photographing it digitally. The curtains on the backdrop are seen through portions of the radiograph as a fine checker pattern. Although this was an inadequate X-ray referral the left-sided effusion is clearly visible. **(B)** Pediatric X-ray referral from the Collegium Telemedicum platform for MSF. The radiograph is of relatively poor quality as it is underinspired and the patient is markedly rotated to the left. In addition, the image has not been converted to gray scale. The right middle lobe air-space disease is visible but difficult to distinguish from density at the bases probably the result of poor inspiration. It is also difficult to comment on the presence of lymphadenopathy due to

## **E-MAIL TELEREADING CONTROLLER AND READER FOR WFPI/KHAYELITSHA PROJECT**

The World Federation of Pediatric Imaging (WFPI) pilot telereading program was created to provide opinions on pediatric X-rays from the Khayelitsha District Hospital in the Western Cape of South Africa, in 2012. As outreach chairman for the society, I acted as manager of teleradiology for a 7-month period. The hospital itself is located in an informal township in the "Cape Flats" serving a population of 406,779 (2005), comprising fewer than 7% adults over 50 years old. Over 40% of the residents are younger than 19 years of age.

This is a new hospital where family physicians supervised by the district pediatrician care for children. Approximately onethird of the 1,000 annual imaging examinations at the hospital are pediatric X-rays. Prior to the pilot teleradiology project, radiographs were interpreted by the physicians. The X-ray equipment

the vascular crowding, again due to under inspiration. **(C)** A radiograph referred from an MSF site in Cambodia is of higher quality, probably owing to the digital equipment available, which also avoids the quality losses from photographing or scanning hard copies. Note that positioning quality is improved with older patients but that on this occasion the right costophrenic angle has been cut off. **(D)** A digital X-ray referral from Khayelitsha District Hospital involves direct conversion of a DICOM image to JPEG for limiting file size and e-mailing through a control telereader to an expert pediatric radiologist. On this radiograph, a right lower lobe air-space disease process was reported but there were no features identified to suggest tuberculosis. The radiographic quality and labeling is excellent, as there are professionally trained radiographers working on-site.

is new, and the on-site radiographers are well trained and produce high-quality radiographs.

As control teleradiologist, I distributed pediatric X-ray referrals to radiologists from a network of up to 50 volunteer pediatric radiologists. All volunteers spoke English and hailed from 17 different countries (USA, South Africa, Brazil, China, India, Colombia, Pakistan, Spain, the UK, Argentina, Bolivia, Cuba, Sri Lanka, Australia, Panama, New Zealand, and Italy).

A total of 555 referrals and 1,106 radiographs were submitted for teleradiology opinion during the period 26 July 2012 to 3 March 2013. The majority of these were chest radiographs (75%), mainly for the evaluation of tuberculosis (**Figure 1D**). I reported 76 referrals, often having to fill in for a non-responding volunteer. In addition, I trained the clinicians on-site in the interpretation pediatric radiographs for the diagnosis of TB on two occasions (**Figure 2A**).

**FIGURE 2 | Training of non-radiologists to interpret radiographs is part and parcel of telereading support and capacity development**. **(A)** A large group of family physicians participating in a pre- and post-training, pediatric X-ray reporting, self-assessment session, hosted by the visiting outreach radiologist from the South African Society of Pediatric Imaging acting (under the auspices of the World Federation of

Pediatric Imaging). This program was run twice in parallel with the pilot telereading program, intended to increase capacity for X-ray interpretation on the ground. **(B)** Point of care mediastinal sonography for the diagnosis of tuberculosis being taught to a non-radiologist by a representative of Imaging the World in co-operation with the World Federation of Pediatric Imaging.

To ensure the sustainability of this program, an "institutional buddy" system is being tested in conjunction with Stanford University. Radiology opinions are provided for specific physician requests through e-mail. Stanford University through the Lucille Packard Children's Hospital offers the service through its residents who provide opinions under the supervision of an experienced consultant. Only six radiographs have been reported so far in the first few months. Stanford University benefits by providing its residents with exposure to developing country pathology. This symbiotic relationship may prove to be the key to sustainability. The WFPI hopes to encourage other university hospitals to engage in such reciprocal systems.

## **WFPI TELEREADING OF CT SCANS FOR THE INDIRA GANDHI CHILDREN'S HOSPITAL PILOT**

Lack of sub-specialist pediatric imaging expertize on-site, the availability of a CT scanner and the significant childhood population visiting the Indira Gandhi Children's Hospital made this a feasible telereading pilot project for the WFPI. We faced several challenges in setting up teleradiology at this site. Lack of proximity of a WFPI member and the geographical distance between the contributors made the setup complex. Meetings are also difficult to schedule because of time zone differences, and communications are primarily through e-mail. We also needed more than e-mail as a platform for interpreting CT scans because of the large amount of images.

A telereading digital platform was created, but it took a long time to get the referring site and WFPI volunteers to install the software. The developers of the software have been helpful but they are distant to the site and are sometimes naïve to the restrictions and obstacles faced locally. The installation of the software requires administrative privileges and the information technology infrastructure of the department must be known for configuring proxy servers, etc. The volunteers' information technology departments did not allow software installation in some instances, which meant that volunteers had to interpret the studies on their own personal computers. There are also cost implications with a current fee of 1.00C (US \$1.36) per uploaded study. Even though some cases have been referred successfully and reports have been provided, this telereading project is stuttering along. Lessons have been learnt regarding requirements for setting up new telereading software and with regard to having personnel on the ground to troubleshoot.

## **COST-EFFECTIVENESS**

Most sustainable teleradiology programs have employed low-cost systems using "low-technology" methods (5). E-mail consultation is a simple solution that has proved to be effective, useful, and acceptable (2) with precedent in the successful Swinfen Charitable Trust projects operation over many years (3). E-mail is not recommended, however, as it is an unsecure, and the delivery mechanism cannot be guaranteed.

Zanaboni and Wootton identified 37 cross-border telemedicine services that used store and forward technology such as e-mail. This was the commonest form of telemedicine, using the Internet and a digital camera (3). An MSF project in Malawi used digitization of X-rays by photographing them using a digital camera and transmitting them electronically to a radiologist initially via e-mail (and subsequently through a web-based telemedicine service – Médecins Sans Frontières/Swinfen Charitable Trust). Reports were returned by the radiologist in the United States to the MSF physician free of charge by e-mail or entry on the telemedicine web site. Teleradiology changed patient management in some cases by reducing the time to a definite diagnosis and

preventing misdiagnosis (1) without measurable cost. Another project in Ethiopia also used photographs of X-rays with a camera, "home-made"open source telemedicine software, and ran between 2004 and 2006 in 10 health care sites. This project was considered unsuccessful, as it was not sustainable. The authors noted problems common to Sub-Saharan African countries – low bandwidth, slow connections, and high service charges (2).

"Low-cost" digital X-ray devices are now available for lowresource settings, and these can significantly improve the quality of local X-ray images. Digital X-ray is a solution for low-resource countries because it eliminates film development and processing, and enables teleradiology (6). One project in Angola implemented digital radiology and provided teleradiology reports free of charge, by e-mail. The cost of purchasing the equipment in 2010 was 26,660 US\$. It was calculated that the initial cost was paid back in 2 years by eliminating the need for expensive films and reagents (6).

## **BARRIERS TO THE PRACTICE OF TELERADIOLOGY**

The major limiting factors for many countries are low bandwidth, slow Internet connections, and high Internet service charges (2). To combat this, JPEG compression is used to decrease image file size but this in turn leads to poorer image quality. Nonetheless, several studies have demonstrated that JPEGs, such as used in a number of MSF projects, obtained by digital photography of radiographs are sufficient for diagnosis in most instances (1).

Language is cited as a potential limitation (5) as it can prevent teleradiologists from understanding key information and prevent the requesting physician from understanding the opinions offered. One solution to this is the use of an international panel of volunteers from all over the world as per the WFPI volunteer network.

Another limitation is the quality of the images referred for interpretation, because the radiologist is distant from the site where the imaging is performed and has little influence over the technologists performing the studies. For teleradiology to be effective, quality standards must be maintained (7). The image quality of hard copy radiographs is affected by factors such as poor equipment, substandard materials, and the inherent nature of the screen-film technology (7). The MSF quality assurance program found that those sites using film and chemistry for X-ray imaging demonstrated significantly more non-diagnostic images than those sites with digital imaging. Inadequate exposure was the most frequent problem (68%, *n* = 143). In all, 24% (*n* = 51) of film images demonstrated artifacts compared with 1% (*n* = 7) of digital images. The five MSF sites performing poorest for quality were all ministry of health facilities using film and chemistry (7).

## **THE CHAIN – ANY BREAK IN A LINK BREAKS THE SYSTEM**

The quality of radiographs sent for a teleradiology opinion can be adversely affected by any part of a long chain of events. Programs converting hard copy into digital files via digital cameras create additional technical considerations. There is limited opportunity to improve the quality of a hard copy image if the primary radiograph is poor (6). Digital technology allows for a wider margin of radiographic error, makes post processing improvements possible, is simpler to use, obviating the need for development and processing, has less inherent artifact, and also simplifies the practice of

teleradiology because images can be directly electronically converted to JPEG format (6). With a shorter chain of events, there are fewer possibilities for error, making this more desirable for limited resource settings.

An effective teleradiology operation also requires appropriately qualified radiologists to be available, and (usually) a mechanism for distributing incoming cases to the most appropriate specialist. This requires a vigilant teleradiology manager who has knowledge of the available human resources and understanding of the different skills of each radiologist.

The strength of the teleradiology chain is equivalent to its sustainability, i.e., the program "must be adopted into everyday practice and continue to function with high activity levels after any pilot funding runs out" (6). Success also depends on suitable governance, effective policy development, human resource management, and capacity building (2). Radiologists familiar with the local environment are a pre-requisite for project survival beyond the initial pilot phase (3).

## **FUTURE RESEARCH**

Collecting and evaluating data from pilot projects are essential for future planning.

Important information includes the proportion of all pediatric examinations sent for teleradiology, the reasons for referral, the elapsed time from request to receipt of opinion, the number of requests not responded to, the effect on patient management, the volunteer teleradiologist workload, number of volunteers who left the project, and the image quality of the JPEGs. It is also extremely important to know the costs. Such research requires that projects have adequate mechanisms for capturing data.

An extension of telereading that requires more research is the interpretation of ultrasound imaging performed by nonradiologists. Point of care sonography, especially for diagnosis of TB and pneumonia in children, is a novel way of overcoming the expense, complexity, and radiation risk associated with X-ray imaging (8). This is in addition to other indications for assessing the kidneys, abdomen, pleural effusions, and intracranial compartment in neonates. Ultrasound is ideally suited for pediatric imaging at the point of care. However, because it is operator-dependent, operators often pay little attention to routine anatomical image acquisitions and labeling and there has been little consideration for telereading. With a little effort to standardize imaging and train on-site personnel in orientation and technique, opinion of ultrasound images through telereading is possible. Standardized imaging protocols already exist for trauma (FAST), abdominal HIV, and TB (FASH) and even cranial ultrasound, but the thrust has been for the person at the point of care to make a focused assessment for immediate management or triage (8).

Imaging The World (ITW) has paved the way for telereading ultrasound imaging performed by personnel who have no anatomical knowledge but understand surface landmarks. These "sonographers"produce standardized sweeps based on surface landmarks that are compressed and sent over the Internet for telereading. WFPI has partnered with ITW to produce pediatric protocols. In addition, a research project testing the diagnostic capability of the telereaders' sweep of the mediastinum against an expert free hand sonographer is in progress in Cape Town South Africa for the diagnosis of TB in children (**Figure 2B**). This research based at the Red Cross Children's Hospital (University of Cape Town) is being expanded for the diagnosis of pneumonia. The research possibilities of telereading for point of care sonography are extensive and are relatively safe due to the nature of sonography. However, image compression techniques and free telereading platforms need to be researched further to meet the limitations of bandwidth and Internet speeds in developing countries.

## **CONCLUSION**

Although teleradiology is a viable option to alleviate radiologist shortages in underserved areas, there are many challenges to designing an adequate teleradiology chain. Teleradiology is possible through simple e-mail of JPEG images or through more sophisticated means and can assist doctors working in developing countries with expert support to manage sick children. Pediatric radiology volunteers have a responsibility to examine further avenues of telereading for point of care sonography through research and training programs.

## **REFERENCES**


**Conflict of Interest Statement:** The author declares that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 14 May 2014; accepted: 06 August 2014; published online: 21 August 2014. Citation: Andronikou S (2014) Pediatric teleradiology in low-income settings and the areas for future research in teleradiology. Front. Public Health 2:125. doi: 10.3389/fpubh.2014.00125*

*This article was submitted to Public Health Education and Promotion, a section of the journal Frontiers in Public Health.*

*Copyright © 2014 Andronikou. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

## The development of a multilingual tool for facilitating the primary-specialty care interface in low resource settings: the MSF tele-expertise system

#### **Laurent Bonnardot 1,2\*, Joanne Liu3,4, ElizabethWootton<sup>5</sup> , Isabel Amoros <sup>6</sup> , David Olson<sup>7</sup> , SidneyWong<sup>8</sup> and RichardWootton9,10**

<sup>1</sup> Médecins Sans Frontières, Paris, France


#### **Edited by:**

Philip Baba Adongo, University of Ghana, Ghana

#### **Reviewed by:**

Wilma Alvarado-Little, AlvaradoLittle Consulting LLC, USA Xiaoguang Ma, University of South Carolina, USA

#### **\*Correspondence:**

Laurent Bonnardot, Fondation Médecins Sans Frontières, 8 Rue Saint Sabin, Paris 75011, France e-mail: lbonnardot@gmail.com

In 2009, Médecins Sans Frontières (MSF) started a pilot trial of store-and-forward telemedicine to support field workers. One network was operated in French and one in English; a third, Spanish network was brought into operation in 2012. The three telemedicine pilots were then combined to form a single multilingual tele-expertise system, tailored to support MSF field staff. We conducted a retrospective analysis of all telemedicine cases referred from April 2010 to March 2014.We also carried out a survey of all users in December 2013. A total of 1039 referrals were received from 41 countries, of which 89% were in English, 10% in French, and 1% in Spanish. The cases covered a very wide range of medical and surgical specialties. The median delay in providing the first specialist response to the referrer was 5.3 h (interquartile range 1.8, 16.4). The survey was sent to 294 referrers and 254 specialists. Of these, 224 were considered as active users (41%). Out of the 548 users, 163 (30%) answered the survey. The majority of referrers (79%) reported that the advice received via the system improved their management of the patient. The main concerns raised by referrers and specialists were the lack of support or promotion of system at headquarters' level and the lack of feedback about patient follow-up. Because of the size of the MSF organization, it is clear that there is potential for further organizational adoption.

**Keywords: telemedicine, telehealth, developing countries, tele-expertise, multilingual network**

## **INTRODUCTION**

Médecins Sans Frontières (MSF) is an international, independent, and medical humanitarian organization that responds to emergency situations and provides medical assistance to people in need affected by armed conflict, epidemics, natural disasters, and exclusion from healthcare (1). A defining characteristic of the organization is its innovation (2). Over the years, MSF has developed considerable expertise in pioneering new technology for resource-limited settings in different fields, such as medical (e.g., automated TB diagnostic testing (GeneXpert), malaria Rapid Diagnostic Test) or logistical (e.g., inflatable hospitals with operating theaters, oxygen concentrators, vaccination kit).

It is not surprising, then, that MSF should take advantage of new information technology to improve the quality of health care for patients in low-resource settings. The work in question began in 2009, when MSF started a pilot trial of two telemedicine systems to support field workers. One was operated in French and one in English; a third, Spanish system was brought into operation in 2012. They were established initially in collaboration with the Swinfen Charitable Trust (3). In late 2013, the three telemedicine pilots were combined into a single multilingual system, using technology based on the Collegium Telemedicus system (4). Because of the constraints of MSF operations (e.g., legal, confidentiality, reporting), the multilingual system was established on a secure web server of its own, telemed. msf.org.

The product of this 4-year development period is a teleexpertise system, tailored to support MSF field staff. It is based on a highly secure web messaging system (see **Box 1**). It aims to facilitate the primary-specialty care interface by allowing a primary care physician to obtain an expert second opinion about a difficult clinical problem within a few hours.

The aim of the present study was:


#### **Box 1 The MSF tele-expertise system**.

#### **Purpose**

A tool for use in the field to improve access to specialized clinical advice. It is available in English, French, and Spanish.

#### **Workflow**


## **Method of operation**

A secure, web-based messaging system. Confidentiality is ensured by removing any identifying patient data. Email is only used for notifications (i.e., that a case has been submitted or an answer received) and for advisory messages (e.g., login reminder).

#### **Example**

Clinical case sent through MSF tele-expertise system.

## **MATERIALS AND METHODS**

We conducted a retrospective analysis of all cases referred from April 2010 to March 2014. Information relating to the cases was extracted from the database of the tele-expertise system. Ethics permission was not required, because patient consent to access the data had been obtained and the work was a retrospective chart review conducted by the organization's staff in accordance with its research policies.

We also carried out a survey of all users in December 2013. The survey contained 50 questions. These were closed-ended, multiple-choice, and scale type questions, and open-ended questions. The questions were established after literature research combined with qualitative data collection (in-depth interview and participating observation). The survey was tested on three referrers and three specialists, in English and in French. After the pilot testing, the survey was sent to all referrers and specialists registered in the database, regardless of whether or not they were active (i.e., had logged in and sent or answered cases). Versions of the survey were made available in French and English. Web-based software (https: //www.surveymonkey.com/) was used for collecting the data.

Data were examined with the usual methods for quantitative analysis, while the results of the open-ended questions were processed in a qualitative way. The present paper reports a preliminary analysis of the survey results.

## **RESULTS**

## **DEVELOPMENT OF THE NETWORK**

Over a 4-year period, the tele-expertise system evolved from separate, single-language telemedicine networks to an integrated, multilingual system. This encompassed:

	- 1 in each language (English, French, Spanish);
	- 1 within each of the 5 MSF operational centers;
	- 1 for radiological cases
	- 2 software engineers (part-time).

## **SYSTEM PERFORMANCE**

During the 4-year study period, the caseload rose in the first 2 years and subsequently stabilized at about 1–2 cases/day (**Figure 2**). The peak was mainly the result of radiology cases submitted from a single hospital in the Central African Republic, which had no radiological expertise available on-site. Fluctuations were mainly related to specific implementation episodes and to promotion in presentations to the organization's staff.

A total of 1039 referrals were received from 41 countries (**Figure 3**), of which 89% were in English, 10% in French, and 1% in Spanish.

The majority of the case allocations were done by four casecoordinators (93%). The median delay in allocating a new case was 0.5 h (interquartile range, IQR 0.17, 1.9). The median delay in providing the first specialist response to the referrer was 5.3 h (IQR 1.8, 16.4).

Two-thirds of the cases (66%) required a single allocation (also known as a query) to produce a specialist response, while onethird of the cases required more than one allocation. The mean number of allocations per case was 1.51 (**Figure 4**). The median number of messages per case was 4 (IQR 3, 6).

## **CASE CHARACTERISTICS**

The cases covered a very wide range of medical and surgical specialties (**Table 1**). The most common type of referral was for radiology (44%), which reflected the difficulties being experienced at a small number of hospitals in sub-Saharan Africa.

The majority of the cases were submitted by relatively few referrers (see **Figure 5**). For example, 80% of cases were submitted by only 10% of the referrers.

Similarly, the majority of queries were answered by relatively few specialists (see **Figure 6**). For example, 80% of queries were sent to only 16% of all specialists.

## **SURVEY RESULTS**

The survey was sent to 294 referrers and 254 specialists. Of these, 224 were considered as active users (41%). Out of the 548 users,163 (30%) answered the survey. The survey was completed reasonably promptly by the majority of the respondents: 70% questionnaires were completed within 6 days. Responses from French and English users were analyzed together.

The survey results were generally positive (**Tables 2A,B**) demonstrating a high level of user participation, despite the questionnaire being rather long (50 questions).

The main features of the users and their IT habits are shown in **Table 3**. Answers to questions related to satisfaction and the benefits of system use are shown in**Tables 3** and **4**. Although many users skipped this part of the survey, this was mainly because the questions could not be answered unless the respondent had actually used the tele-expertise system.

A summary of the commonly-occurring referrer and specialist comments made in response to the open-ended questions are shown in **Tables 5** and **6**. The main concerns raised by referrers and

specialists were the lack of support or promotion of system at headquarters' level and the lack of feedback about patient follow-up.

## **DISCUSSION**

Médecins Sans Frontières has previously used both store-andforward and real-time telemedicine (5, 6). Although the real-time telemedicine work was considered successful, the requirement for good quality Internet connections makes real-time telemedicine much more expensive than store-and-forward work. Cost is crucial in the humanitarian context or in places which have very few resources. Indeed, the consequences of wastage that would

#### **Table 1 | Types of queries**.


have little effect on health care in high income countries can have a profound impact in low-resource settings. Store-and-forward telemedicine certainly has disadvantages in comparison with realtime telemedicine – principally, the interaction between the parties is not as immediate – but it also has considerable advantages: it is cheaper and it is easier to organize. Thus in a low-resource setting, store-and-forward telemedicine is inherently more likely to be sustainable.

The present review shows that the experience of the MSF teleexpertise system is generally positive. At the time of writing, it is in its fifth year of operation, and as is well-known, many telemedicine projects fail to survive beyond their initial set-up phase (7). Another positive sign is that the referrers who sent cases continued to do so, an objective demonstration of their satisfaction with the system and its value to them.

Comments made by the volunteer specialists suggest that they were highly motivated and frequently expressed frustration about not getting enough cases. It is clear that the positive image of MSF worldwide has been a key factor in recruiting and keeping motivated our specialist volunteers.

The creation of the network, which was set up initially in a few months, is itself a kind of achievement. It reflects the global footprint of MSF, its considerable field expertise, and its multilingualism and multiculturalism: 550 users, 74 countries connected,

#### **Table 2 | (A) Specialist responses; (B) referrer responses**.


<sup>a</sup>Average rating: scale 1 = no, 2 = a little, 3 = moderately, 4 = a lot.

<sup>b</sup>Percentage of total answering yes (56).

<sup>c</sup>Average rating: scale from 0 = not happy at all to 10 = extremely happy with it.

<sup>d</sup>Answer yes = categories 2–4 below.

and tele-expertise available in three languages from specialists with significant field experience.

Based on the user survey, it is clear that the tele-expertise system is easy to use and provides clinically useful diagnostic and management advice to clinicians in the field. The majority of referrers (79%) reported that the advice received via the system improved their management of the patient. In contrast, only about half of the specialists (45%) felt that the advice they had given would improve patient management while another half did not know/were not able to answer (unknown). The same phenomenon was reported in a recent survey of the users of the Swinfen telemedicine system (8). In the present study, the explanation may be that since many specialists had not answered any case, they were not in a position to comment on potential improvement.

Finally, if objective improvement in patent management remains to be demonstrated from the patient point of view, it is clear that there is a precious educational value for the referrers who take full advantage of expert advice and experience to assist them in overcoming their professional isolation.

## **LIMITATIONS OF THE STUDY**

The main limitation of the present study is that it was retrospective, and there was no control system to compare it with. On the other hand, the survey questionnaires were completed by both users and non-users of the system,which reduces the bias inherent in surveys that are only completed by system users.

The response rate to the survey was not as high as would be expected in an online survey of doctors in industrialized countries,

#### **Table 3 | Main features of the user profile**.


For the yes/no questions, the value in brackets represents the majority response. For the multiple-choice questions, the value shown is the majority response. NA, not applicable or not asked.

#### **Table 4 | Comparison of referrer and specialist opinions about the benefits and satisfaction with the system**.


<sup>a</sup>Scale 1 = no, 2 = a little, 3 = moderately, 4 = a lot.

<sup>b</sup>Average rating: scale from 0 = not happy at all to 10 = extremely happy with it.

#### **Table 5 | Summary of referrer comments (open-ended questions)**.


#### **Table 6 | Summary of specialist comments (open-ended questions)**.


where response rates of 50–60% can be achieved. However, in the context of an online survey of telemedicine doctors in lowresource settings, the response rate was reasonable. For comparison, a previous survey of an HIV telemedicine network had a response rate of only 19% (9). The dangers of a low response rate are non-response bias (if the answers of respondents differ from the potential answers of those who did not respond) and response bias (if respondents tend to give answers that they believe that the questioner wants). Since we are not aware of the opinions of the non-responders, this may represent a potential source of bias in the present work.

#### **LESSONS LEARNED**

The two main lessons learned concern the uneven pattern of system usage and the relative lack of referrer feedback:

## **Uneven pattern of usage**

Although there are 550 registered users, only about half of them are active, i.e., have logged in and sent or answered cases. We believe that this is typical of large telemedicine systems of this type, but there appear to be few published reports for comparison. Furthermore, the distribution of activity among the active users was very uneven, e.g., 80% of cases were submitted by only 10% of the referrers, and 80% of queries were sent to only 16% of all specialists. This uneven pattern of usage may lead to the demotivation of specialists who agree to answer cases, but do not subsequently receive referrals. The uneven pattern of referrals may be a consequence of limited communication and promotion of the system by MSF, with little briefing of staff before their deployment to the field; both reflect a lack of political support to embrace telemedicine. In the future, positive attempts must be made to engage all users.

#### **Lack of referrer feedback**

Feedback from the referrer about patient follow-up is crucial for quality improvement and is necessary to keep the volunteer specialists informed about cases that they have advised on. The lack of feedback from referrers may also lead some specialists to lose interest in continued participation. Almost all specialists request follow-up after a teleconsultation (52% considered followup desirable and 47% considered it necessary or mandatory), and most referrers acknowledge a willingness to provide it. The reasons for the relative lack of follow-up data are probably not due to an unwillingness to provide it by the referrers. The stated reasons include a lack of time and a feeling that it was unnecessary. In addition, it is the nature of MSF operations in conflict zones and other resource-limited settings that patients are often seen in hospital, treated, and then disappear, not being available for subsequent follow-up to take place. Despite these practical difficulties, we have recently established a system by which follow-up requests are sent to the referrer automatically by email after a predetermined interval. This may improve the feedback in future.

## **CONCLUSION**

After 4 years of development,MSF has put into place a multilingual tele-expertise system to support workers in the field. User surveys confirm that the system provides helpful advice, which has a positive effect on patient outcomes. It is reliable and efficient. It improves patient management, has educational value for those involved, and reduces isolation for the referrers. Because of the size of the MSF organization, it is clear that there is potential for further organizational adoption. This will depend on political support from within the organization itself.

## **ACKNOWLEDGMENTS**

We thank the volunteer case-coordinators and specialists,who represent the cornerstone of the system. We are particularly grateful to Jean Rigal, Jean-Hervé Bradol, Marie Pierre Allié, Will Wu, Olivier Steichen, Christian Hervé, Ondine Ripka and the MSF legal department, Leslie Shanks, Katherine Gottwald, RaghuVenugopal, Pedro Pablo Pablo, Carme Baraldes, Eric Comte, Bertrand Draguez, Jean-Paul Jemmy, Saskia Spijker, Judith Herrera, Annette Heinzelmann, Micaela Serafini, Jarred Halton, Elisa Salvado, and all our field referrers.

## **REFERENCES**


paediatric care in a conflict setting in Somalia? *Trop Med Int Health* (2012) **17**(9):1156–62. doi:10.1111/j.1365-3156.2012.03047.x


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 14 June 2014; paper pending published: 15 July 2014; accepted: 07 August 2014; published online: 26 August 2014.*

*Citation: Bonnardot L, Liu J, Wootton E, Amoros I, Olson D, Wong S and Wootton R (2014) The development of a multilingual tool for facilitating the primary-specialty care interface in low resource settings: the MSF tele-expertise system. Front. Public Health 2:126. doi: 10.3389/fpubh.2014.00126*

*This article was submitted to Public Health Education and Promotion, a section of the journal Frontiers in Public Health.*

*Copyright © 2014 Bonnardot, Liu, Wootton, Amoros, Olson, Wong and Wootton. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

## Teleradiology usage and user satisfaction with the telemedicine system operated by Médecins Sans Frontières

#### **Jarred Halton<sup>1</sup>\*, Cara Kosack <sup>1</sup> , Saskia Spijker 1,2, Elizabeth Joekes <sup>3</sup> , Savvas Andronikou<sup>4</sup> , Karen Chetcuti <sup>5</sup> , William E. Brant 6,7, Laurent Bonnardot 8,9 and RichardWootton10,11**

<sup>1</sup> Médecins Sans Frontières Diagnostic Network, Amsterdam, Netherlands


<sup>9</sup> Fondation Médecins Sans Frontières, Paris, France

<sup>10</sup> Norwegian Centre for Integrated Care and Telemedicine, University Hospital of North Norway, Tromsø, Norway

<sup>11</sup> Faculty of Health Sciences, University of Tromsø, Tromsø, Norway

#### **Edited by:**

Sue Ellen Levkoff, University of South Carolina, USA

#### **Reviewed by:**

Ruitai Shao, World Health Organization, Switzerland Suzanne M. Hardeman, University of South Carolina, USA

#### **\*Correspondence:**

Jarred Halton, Médecins Sans Frontières, Plantage Middenlaan 14, Level 4, Amsterdam 1018 DD, Netherlands e-mail: jarred.halton@ amsterdam.msf.org

Médecins Sans Frontières (MSF) began a pilot trial of store-and-forward telemedicine in 2010, initially operating separate networks in English, French, and Spanish; these were merged into a single, multilingual platform in 2013. We reviewed the pattern of teleradiology usage on the MSF telemedicine platform in the 4-year period from April 2010. In total, 564 teleradiology cases were submitted from 22 different countries. A total of 1114 files were uploaded with the 564 cases, the majority being of type JPEG (n = 1081, 97%). The median file size was 938 kb (interquartile range, IQR 163–1659). A panel of 14 radiologists was available to report cases, but most (90%) were reported by only 4 radiologists. The median radiologist response time was 6.1 h (IQR 3.0–20). A user satisfaction survey was sent to 29 users in the last 6 months of the study. There was a 28% response rate. Most respondents found the radiologist's advice helpful and all of them stated that the advice assisted in clarification of a diagnosis. Although some MSF sites made substantial use of the system for teleradiology, there is considerable potential for expansion. More promotion of telemedicine may be needed at different levels of the organization to increase engagement of staff.

**Keywords: telemedicine, telehealth, teleradiology, LMICs**

## **INTRODUCTION**

Médecins Sans Frontières (MSF), a medical humanitarian emergency organization, operates in resource-limited settings where there are often difficulties in accessing good quality medical imaging services, such as X-ray and ultrasound. Unfortunately, there is little published information about the shortages of imaging equipment and the availability of qualified radiologists to interpret the images (1). In 2012, MSF was operating at approximately 750 locations (field sites) globally, although most did not have X-ray or ultrasound imaging facilities available on site (2). Of the sites with X-ray and ultrasound imaging available, some were MSF installed and operated, while others used local ministry of health services.

In 2010, MSF began a pilot project to provide its field sites (places where MSF provides health care services) with access to a network of specialists through a store-and-forward telemedicine platform based on the Collegium Telemedicus system (3). The system initially operated separate networks for English, French, and Spanish users, but these were merged into a single, multilingual platform in 2013. The MSF telemedicine (tele-expertise)

system provides field doctors with access to a very wide range of specialists, including radiologists.

#### **HOW THE SYSTEM WORKS**

The telemedicine system requires field site users to log onto a secure website to submit a case that includes a brief patient history and anonymized images from different modalities. Typically, these images are from general X-ray or ultrasound examinations.

Two different types of images can be sent for teleradiology: files in DICOM format (Digital Imaging and Communications in Medicine) or files in JPEG format. Sites with access to digitally acquired images are able to upload DICOM or JPEG files exported from the medical imaging device. Sites using traditional methods of X-ray development, i.e., film and chemistry developing, are provided with a protocol for creating a digital file by photographing X-rays of acceptable quality (4, 5). The protocol for digitizing X-rays requires a digital camera of at least 3.5 Mpixel resolution, with optical image stabilization (settings: compression high, flash off, autofocus, exposure compensation set manually to +1.3 EV

for chest X-rays, 0 for other anatomical areas) fixed to a tripod placed 70 cm from the light box perpendicular to the film with extraneous light blocked from the outer edges of the X-ray (4, 5).

Once a case has been submitted, an email prompt is sent automatically to the coordinators of the system who can then log on to the website and allocate the case. The allocation of a case to a radiologist for consultation is referred to as a query. Some cases require allocation to more than one radiologist, e.g., if that person is unavailable for some reason. Therefore, there are more queries than cases. An email prompt is sent to the radiologist once a case is allocated to them, and a subsequent email prompt is sent to the field site when the radiologist has submitted their findings. All specialists' findings are purely advisory, and the final decision on patient management remains with the clinician in the field. In all cases, images, findings, and correspondence are stored securely on the telemedicine platform for ease of future reference. Patient data and images are stored on the secure website but are not included in email messages for reasons of confidentiality.

## **OBJECTIVE**

The aim of the present study was to assess teleradiology usage on the MSF telemedicine platform since its inception. Our hypothesis was that there would be an overall growth in teleradiology usage.

## **MATERIALS AND METHODS**

We conducted a retrospective analysis of all cases sent for radiologist consultation from April 2010 to March 2014 from data extracted from the MSF telemedicine system. Ethics permission was not required, because patient consent to access the data had been obtained and the work was a retrospective chart review conducted by the organization's staff in accordance with its research policies.

We collated the extracted data on a spreadsheet (Excel 2007, Microsoft) and analyzed the number and origin of cases, turnaround times, the number of cases read by participating radiologists, the number of queries per case, and overall usage. Any cases apparently representing statistical outliers were reviewed individually to ascertain the reason.

Starting in October 2013, requests were sent to field staff referring cases to complete a user feedback questionnaire. The requests were sent 21 days after the referral was made. The questionnaire contained 12 questions, part of a larger study to be reported elsewhere. The present work considers the two questions relating to user satisfaction:

(1) Did you find the advice helpful? (2) If YES, did it- (tick any that apply) - Clarify your diagnosis - Assist with your management of the patient - Improve the patient's symptoms - Improve function - Any other reason? Please specify **RESULTS NUMBER OF CASES**

In total, 564 teleradiology cases were sent in the 4-year period from April 2010 to March 2014. The first case was submitted in

June 2011, so the mean referral rate from that time was approximately 16 cases per month. The maximum number of cases sent was 64, in October 2012. There was no clear pattern of overall growth in usage during the study period.

## **IMAGES**

A total of 1114 files were uploaded with the 564 cases submitted. The majority of the uploaded files were of type JPEG (*n* = 1081, 97%); there were 18 compressed files (.zip), which were also mainly JPEG type (**Table 1**). The average number of uploaded files per case was 1.98 (range 1–16). The median file size was 938 kb (IQR 163–1659).

## **NUMBER OF QUERIES**

In total, there were 661 queries from the system for a radiologist consultation. This represented an average of 1.2 queries per case. The number of queries varied between radiologists from 0 to a maximum of 32 per month for the busiest, see **Figure 1**.

## **ORIGIN OF CASES**

A total of 564 cases were sent from 22 different countries (see **Table 2**). The majority of cases, 69% (388/564), came from two countries: Central African Republic and Malawi.

## **TURNAROUND TIME**

The median delay in allocating a case by the system coordinators was 0.4 h (IQR 0.1–1.3). The median radiologist response time, which includes the delay in allocation, was 6.1 h (IQR 3.0–20), which is based on 563 cases. One additional case was allocated, but left unanswered and was excluded from the turnaround time results.

## **RADIOLOGISTS**

A total of 14 radiologists were available through the system and during the study period 12 radiologists were sent at least one case.

## **Table 1 | Contents of the compressed files**.


Four radiologists were allocated 90% (507/564) of all cases, with radiologist B allocated the majority, 64% (361/564), **Figure 2**.

## **USER SATISFACTION**

Between 27 October 2013 and 31 March 2014, a total of 29 requests for completion of the survey were sent to referrers who had submitted radiology cases. A total of eight completed surveys were received (28% response rate). Most respondents (7/8) found the radiologist's advice helpful and all (6/6) stated that the advice assisted in clarification of a diagnosis, see **Table 3**.

## **DISCUSSION**

## **NUMBER OF CASES**

Teleradiology via the MSF Telemedicine system has been possible since April 2010, although it took over a year for the first teleradiology case to be sent in June 2011. The lack of system usage during that time was probably due to a lack of awareness within field sites and operational centers, the time taken to acquire acceptance of a new technology, and the limited number of field sites with access to both medical imaging facilities and a suitable Internet connection. The increase in cases requesting radiology advice occurred after a focused effort by the MSF Diagnostic Imaging Working Group to promote the service among headquarters and field staff through emails, internal newsletters, presentations at internal scientific events and workshops.

## **ORIGIN OF CASES**

Médecins Sans Frontières has been operating in Malawi and Central African Republic for over 20 years (6, 7). Recent collaborations on the use of teleradiology in Malawi (8) and an X-ray installation with mandatory teleradiology use in Central African Republic probably explain the high number of cases from these countries.

## **USAGE**

Although some field sites made substantial use of the system for teleradiology, there is clearly considerable potential for expansion. There were also fluctuations in the use of the system, which can probably be attributed to changes in personnel at field sites familiar with the telemedicine system, limitations to Internet access in remote locations, and periods of medical equipment repair/maintenance. The pattern of usage for the six sites referring the most cases showed no general trends (see **Figure 3**). A number of sites sent fewer than 10 cases per month with irregular, unexplained peaks, and troughs. The period from April 2012 to January 2013 showed relatively consistent usage with the peak volume of teleradiology cases occurring in October 2012. This coincided with the field site in Central African Republic commencing use of the system, which continued until the end of January 2013. The usage stopped in January 2013 due to the deterioration of the security situation and removal of most expatriate staff from Boguila in the Central African Republic. This also explains in part


\* The small number of cases from France and the Netherlands were submitted by headquarters staff on behalf of field doctors in unidentified countries.

the decreases in total number of cases for the period February 2013 to March 2014.

## **IMAGES**

A quality assessment of all X-rays sent for teleradiology in 2012 for MSF reported a clear superiority in the quality of X-rays originating from sites with computed radiography (CR) compared with traditional film chemistry development (9). This assessment included the MSF Telemedicine system and a separate platform, vRad (Virtual Radiologic Corporation, Eden Prairie, MN, USA), a commercial teleradiology company providing *pro bono* services to MSF.

## **NUMBER OF QUERIES**

The difference between the number of cases (554) and the number of queries (661) was caused by certain cases requiring a second or multiple allocations to different radiologists. This may occur in situations when the radiologist first allocated a case was not available to reply within 24 h, the allocation was made to an incorrect sub specialist (for example, adult images being sent to a pediatric radiologist), or the expertise of more than one radiologist was required for a single case.

During the evolution of the system, detailed information about radiologist availability and sub specialty expertise was acquired to guide coordinators in suitably allocating cases. This has improved the turnaround times.

## **TURNAROUND TIME**

The telemedicine system is available 24 h a day, 7 days a week and cases are submitted, allocated, and reviewed from different countries and time zones around the world. Cases are allocated based on the clinical question raised by the field worker, type of imaging study, sub specialty of the radiologist, and the time zone the radiologist reporting the cases operates from. The system is not designed for life-threatening emergency cases, since it operates in store-and-forward mode.

## **RADIOLOGISTS**

The system has relied heavily on four radiologists who were consulted in the majority of cases (90%). This has been beneficial for continuity and familiarity between the field site staff and the radiologist, and the radiologist's understanding of available resources and typical disease burdens at that site. However, this is also a potential disadvantage in the future, especially if the volume of cases increases, as it may create an over-reliance on a small number of radiologists and loss of interest from other less active radiologists. Recruitment of additional radiologists is hampered by the requirement to understand local disease epidemiology and health care setting. Radiologists working in low resource settings are scarce and less likely to be available for teleradiology support, in addition to their daily workload.

## **LESSONS LEARNT**

Despite the system being available to all MSF field sites with access to medical imaging facilities and despite very few radiologists being present in the field, only a limited number of sites have made regular use the service. There has been no overall trend of increased use of the system. There could be a number of reasons for this, including a lack of knowledge about the availability of teleradiology. A regular program of information about the system might be helpful in the future. Periods of peak usage were directly related to particular sites or individual referrers, rather than a collective increase across numerous sites. Briefing of expatriate staff prior to departure for a field site, and promotion of use by medical advisers


#### **Table 3 | User satisfaction survey results**.

at headquarters, would be beneficial in ensuring the continued expansion of the service.

## **LIMITATIONS**

The system is easy to use for all those involved. However, due to the high turnover of field staff, certain problems have occurred, such as login details (username and password) being lost or forgotten and being re-requested frequently, and images not being uploaded with a case. In addition, there was one unanswered query that was overlooked at the end of a long thread of patient discussion. User satisfaction surveys are a recent addition and only a small number of cases were included.

## **CONCLUSION**

The multilingual telemedicine system has been used successfully by several field sites for teleradiology, with a total of 564 cases submitted, at a mean rate of about 16 cases per month. The median response time was 6.1 h. Most field users found the radiologist's advice helpful and stated that the advice assisted in clarification of a diagnosis. Despite the system being available to all MSF field sites with access to medical imaging facilities, there has been no overall growth in use of the system for teleradiology. More promotion of telemedicine may be needed at different levels of the organization to increase engagement of staff.

## **ACKNOWLEDGMENTS**

We thank all referring physicians who completed the questionnaire, the volunteer case-coordinators, the radiologists, and other specialists upon whom the success of the system relies.

## **REFERENCES**


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 25 July 2014; accepted: 05 October 2014; published online: 28 October 2014. Citation: Halton J, Kosack C, Spijker S, Joekes E, Andronikou S, Chetcuti K, Brant WE, Bonnardot L and Wootton R (2014) Teleradiology usage and user satisfaction with the telemedicine system operated by Médecins Sans Frontières. Front. Public Health 2:202. doi: 10.3389/fpubh.2014.00202*

*This article was submitted to Public Health Education and Promotion, a section of the journal Frontiers in Public Health.*

*Copyright © 2014 Halton, Kosack, Spijker, Joekes, Andronikou, Chetcuti, Brant, Bonnardot and Wootton. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

## Teledermatology in low-resource settings: the MSF experience with a multilingual tele-expertise platform

#### **Sophie Delaigue<sup>1</sup>\*, Jean-Jacques Morand<sup>2</sup> , David Olson<sup>3</sup> , RichardWootton4,5 and Laurent Bonnardot 6,7**


#### **Edited by:**

Michal Grivna, United Arab Emirates University, United Arab Emirates

#### **Reviewed by:**

Ricardo J. Wray, University of Pennsylvania, USA Xiaoguang Ma, University of South Carolina, USA Maria Zolfo, Institute of Tropical Medicine, Belgium

#### **\*Correspondence:**

Sophie Delaigue, Chemin des Bourrely, 13015 Marseille, France e-mail: delaiguesophie@yahoo.fr

**Introduction:** In 2010, Médecins Sans Frontières (MSF) launched a tele-expertise system to improve the access to specialized clinical support for its field health workers. Among medical specialties, dermatology is the second most commonly requested type of teleexpertise. The aim of the present study was to review all MSF teledermatology cases in the first 4 years of operation. Our hypothesis was that the review would enable the identification of key areas for improvement in the current MSF teledermatology system.

**Methods:**We carried out a retrospective analysis of all dermatology cases referred by MSF field doctors through the MSF platform from April 2010 until February 2014. We conducted a quantitative and qualitative analysis based on a survey sent to all referrers and specialists involved in these cases.

**Results:** A total of 65 clinical cases were recorded by the system and 26 experts were involved in case management. The median delay in providing the first specialist response was 10.2 h (IQR 3.7–21.1).The median delay in allocating a new case was 0.96 h (IQR 0.26– 3.05).The three main countries of case origin were South Sudan (29%), Ethiopia (12%), and Democratic Republic of Congo (10%). The most common topics treated were infectious diseases (46%), inflammatory diseases (25%), and genetic diseases (14%). One-third of users completed the survey. The two main issues raised by specialists and/or referrers were the lack of feedback about patient follow-up and the insufficient quality of clinical details and information supplied by referrers.

**Discussion:**The system clearly delivered a useful service to referrers because the workload rose steadily during the 4-year study period. Nonetheless, user surveys and retrospective analysis suggest that the MSF teledermatology system can be improved by providing guidance on best practice, using pre-filled referral forms, following-up the cases after teleconsultation, and establishing standards for clinical photography.

**Keywords: telemedicine, telehealth, dermatology, LMICs, low-resource settings**

## **INTRODUCTION**

Telemedicine is broadly defined as any kind of medical activity where distance is involved (1). Tele-expertise, as defined in the French Public Health Code, is one of the five main areas of telemedicine (see **Table 1**) (2). Telemedicine applications can be divided into two types, according to their mode of information transmission: synchronous (or real time, e.g., videoconferencing) and asynchronous (or store-and-forward, e.g., email).

There is evidence in the literature showing that telemedicine is useful in low-income countries, both for educational and clinical purposes (3). In low-resource settings, there is a chronic shortage of specialists (4), and it has been shown that telemedicine can improve the quality and accessibility of medical care (5) while avoiding costly referrals (6, 7). Telemedicine also has valuable benefits in reducing the isolation of field doctors (8) and facilitating distance education for field health workers who frequently have no other opportunity to access specialized training.

In 2010, Médecins Sans Frontières (MSF) launched a telemedicine project (**Box 1**) with the aim of improving access to specialized clinical support for its field health workers. The MSF tele-expertise network is based on the Collegium Telemedicus (9) design. It uses a web-based messaging system hosted on a secure server, and storeand-forward methods, which appear to be more appropriate (10) than real time systems in resource-limited settings, because the quality of Internet connection and cost is critical in a humanitarian context (11).

<sup>1</sup> Dermatology Department, Hopital Nord, Marseille, France

<sup>2</sup> Department of Dermatology, Sainte Anne Military Hospital, Toulon, France

<sup>3</sup> Médecins Sans Frontières, New York, NY, USA

<sup>4</sup> Norwegian Centre for Integrated Care and Telemedicine, University Hospital of North Norway, Tromsø, Norway

The MSF tele-expertise network has supported a total of 1039 clinical cases, to date, in a wide range of specialties. The three main specialties treated in the MSF telemedicine system are in descending order, radiology, pediatrics, and medical specialties (12). Among the medical specialties, dermatology is the second most common medical specialty after infectious diseases in terms of the number of queries. Thus, dermatology has an important place in the system, which justifies the present analysis.

The aim of the present study was to review all MSF teledermatology cases in the first 4 years of operation. Our hypothesis was

## **Table 1 | Five areas of telemedicine as defined in the French public health code [2]**.


that the review would enable the identification of key areas for improvement in the current MSF teledermatology system.

## **MATERIALS AND METHODS**

We conducted a retrospective analysis of all dermatological cases referred by MSF field doctors to the MSF telemedicine platform from April 2010 to February 2014.

All cases classified by the IT system as dermatological were retrieved automatically (group 1). To be exhaustive, a manual check of the database was then performed by identifying the most active expert profiles (using the function "user case history") and all potential cross specialties such as internal medicine, infectious diseases, pediatric, and ear nose throat (ENT) in order to extract all dermatological cases (group 2).

## **WORKLOAD**

We performed a descriptive analysis involving the assessment of the cases submitted. First, we extracted data about system performance, as follows: the number and language of cases, the median delay in providing the first specialist response, and the median delay in allocating a new case. Then, we extracted information on case characteristics, including the countries of origin of the experts and their specialties, the most common topics, the age of the patients, the number of images per case, and the number of follow-up reports and cases of poor quality information reported by specialists.

## **Table 2 | (A) Specialist responses (response rate 13/26** = **50%), (B) referrer responses (response rate 9/22** = **41%)**.


<sup>a</sup>Scale from 0 = not happy at all to 10 = extremely happy with it.

## **IMAGES**

We also carried out a qualitative analysis of all images uploaded with the cases submitted. The quality of images was scored as poor, average, or good quality by three of the authors selected because they were clinicians. Images were not scored individually but by case, because this was more clinically meaningful. The assessment was made blind and the results were averaged. The score included a technical analysis of the picture based on focus, anatomical perspective, lighting, and composition (primary lesion and overview picture) (13).

## **SURVEY**

In May 2014, we carried out a short-anonymous survey of all users involved in the dermatology cases. The survey design was based on a previous, larger survey, with 50 questions, established after a literature search combined with qualitative data collection. This survey showed that telemedicine was helpful and improved the management of the patient. We wanted to focus some of these questions on dermatological topics. We focused on (i) follow-up because in this large survey it was one of the main lessons learnt and (ii) quality of the referral because the quality of the expertise depends on the quality of the information send (14).

The survey had six questions for the referrers and seven questions for the specialists (closed-ended, opened-ended, scale type, and multiple choice questions). The questions focused on the quality of the referral and follow-up of the patient (**Tables 2** and **3**). Versions of the survey were made available in French and English. Web-based software (https://www.surveymonkey. com/) was used for collecting the data. Responses were anonymous.

## **CONFIDENTIALITY AND SECURITY**

Ethics permission was not required, because patient consent to access the data had been obtained and the work was a retrospective chart review conducted by the organization's staff in accordance with its research policies (15). Before a new case could be submitted, the referrer had to indicate agreement with the statement "I confirm that informed consent has been obtained from the patient about making an E-referral and its consequences."

Photography is an important tool for diagnosis in dermatology. To ensure patient privacy, several safeguards are implemented in the telemedicine system. First, the referrer is required to avoid transmitting any identifying data (e.g., the patient's **Table 3 | (A) Summary of specialist comments (open-ended questions), (B) summary of referrer comments (open-ended questions)**.


name). Second, the coordinator ensures that this is respected when he/she allocates the case. Finally, it is recommended that clinical photographs be anonymized by, for example, putting a black bar over the patient's eyes.

The Collegium Telemedicus system uses secure messaging. Messages are encrypted bidirectionally and are stored on the server; they are only available to the user via a secure SSL connection (10).

## **RESULTS**

## **SYSTEM PERFORMANCE**

During the study period, 65 clinical cases from 24 countries were handled by the system. There was a steady increase in the caseload over the 4 year period (**Figure 1**). Seventy-one percent of the cases were referred in English and 29% in French. No case was submitted in Spanish.

The median delay in providing the first specialist response to the referrer was 10.2 h (IQR 3.7–21.1). The median delay in allocating a new case was 0.96 h (IQR 0.26–3.05). The majority (83%) of the case allocations were done by two case-coordinators.

#### **CASE CHARACTERISTICS**

Most of the cases were focused on diagnosis issues. Among the 65 cases, 43 were tagged as dermatological cases (i.e., group 1) and 22 were cross-specialty cases (i.e., group 2). The three main specialties involved in these cases were infectious diseases (10), pediatrics (8), and internal medicine (4). Examples of these cross-specialty cases are given in **Figures 2**–**4**.

The three main countries of case origin were South Sudan (29%), Ethiopia (12%), and Democratic Republic of Congo (10%). Africa (74%) was the main continent of case origin. There were small numbers of cases from Kenya, Yemen, Haiti, Bolivia, India, Cambodia, and Central African Republic (CAR) (**Figure 5**). The countries of origin of the experts were, in descending order, France (9), Canada (5), the Netherlands (4), USA (2), Australia (2), Peru (1), New Zealand (1), Spain (1),

**FIGURE 2 | Confirmed histoid leprosy**.

**FIGURE 3 | Suspected mycobacterial infection**.

**FIGURE 4 | Neurofibromatosis**.

and the UK (1). Experts were specialized in pediatrics (11), dermatology (5), internal medicine (6) plastic surgery (1), general surgery (1), ENT surgery (1), and infectious diseases (1). Countries of origin of both referrers and specialist are shown in **Figure 5**.

The most common topics treated were infectious diseases (46%), inflammatory diseases (25%), genetic diseases (14%), and tumor diseases (12%) (**Figure 6**). Bacterial and mycobacterial infections were the two main sub-topics of infectious diseases. Slightly more than half of the cases (51%) were pediatric (under 18 years old).

A total of 216 images were uploaded with the 65 cases submitted and were reviewed by three specialists. The majority of the images attached were of type JPEG – Joint Photographic Expert Group – (52 cases, 84%), there were 8 cases with compressed files (zip), 2 had copied their images into a Word document and 3 had no attached files. The median number of images per case was 3 (IQR 2, 5). The median file size was 345 kb (IQR 101–1593). Moreover, in 4 cases of the 65, pictures attached were not properly anonymized (patient's names were mentioned).

The overall quality of the attached pictures were judged by three of the authors as poor quality in 15%, as sufficient for establishing a correct diagnosis in 53% and as good quality in 32%. In addition to that, experts mentioned in 15% of cases in their answers that the pictures sent were of poor quality.

Only 10 (15%) cases had follow-up data. Two reported the death of the patient. The lack of information about patient follow-up was a critical issue and represents a limitation of the analysis.

Fourteen percent of cases were reported as poor quality or not good enough to make a diagnosis.

## **SURVEY RESULTS**

The survey was sent to 41 referrers and 26 specialists involved in the cases. The survey was completed promptly within 6 days by 22 users (33%: 9 referrers and 13 specialists). Proportionally, more experts completed questionnaires than referrers – 50% (13/26) versus 22% (9/41), respectively. Responses from French and English users were analyzed together.

The principal concern raised by referrers and specialists was the lack of feedback about patient follow-up. Very few experts received any follow-up information about the patients for whom they gave a second opinion, although 56% of the referrers managed to obtain follow-up of referred cases. According to referrers, the optimum period for sending a follow-up report was 1 week.

The second concern raised by specialists was the quality of the clinical details and information supplied by referrers. Summaries of the main referrer and specialist comments made in response to the open-ended questions are shown in **Table 3**.

## **DISCUSSION**

The present study reviewed all MSF teledermatology cases in the first 4 years of operation with the aim of identifying areas for improvement. The system clearly delivered a useful service to referrers because the workload rose steadily during the study period. The two main concerns raised by the users were a lack of followup information and the quality of the information provided in a referral.

## **LIMITATIONS OF THE STUDY**

Due to the lack of follow-up and feedback from the field, we were not able to conduct a proper case content analysis in order to assess the overall impact of the system. Although a bigger survey would be better, the aim of the present study was a descriptive analysis. The survey was conducted in order to provide data about improving the system. The chosen topics (feedback and quality of the referrals) were based on existing information. First, the question of feedback was based on another larger survey, which confirmed that lack of feedback is an important source of weakness (12). We wanted to know if this was the case for dermatology. Second, we wanted to assess the quality of the referrals, as it is well established that there is a direct link between the quality of medical records and the quality of healthcare (14).

The high number of experts (26) involved in case management compared to the relatively small number of cases contributed to the heterogeneity of the results and the difficulty in drawing clear conclusions.

Finally, we were not able to compare the telemedicine system to other methods of accessing specialist dermatology advice because there is no structured system within MSF for doing so – only individual practice. There are also very few studies on this topic in low-resource settings with which to compare.

## **USEFUL TOOL**

With more than 15 years' experience, it is now clearly established that this kind of system is reliable, efficient, and easy to use for doctors working in developing countries (16). Like other long-running telemedicine networks delivering humanitarian medical services such as the Africa teledermatology project (17), the MSF telemedicine system confirms that teledermatology is an important area of use and development. There is nothing surprising in this observation as this specialty is mainly based on visual diagnosis and dermatology conditions are often manifestations of underlying illnesses, such as infectious disease, which can have its own specialist input, making dermatology a particularly good fit for tele-expertise.

The increase in the number of referrals over the past 4 years is a sign of vitality and confirms the growing need for such a system. It therefore seems crucial to enhance the MSF telemedicine system to make sure it can absorb this growth while remaining efficient. As there is no other option for obtaining access to a specialist consultant in most of these low resource settings, this kind of system represents a pragmatic and efficient answer to the chronic shortage of specialists. Furthermore, it is worth noting that with a mean delay of 10 h to the first specialist answer, many industrialized countries would be envious of this level of response. This remarkable reactivity from our experts who all work full time in other settings is undoubtedly linked to their motivation in providing assistance to isolated doctors with the shortest delay. But, it is also clear that by dealing with cases out of their usual practice environment, our experts find a personal interest in handling some of the rare and interesting cases referred with professional value both clinical and academic (see for example, **Figure 7**).

**FIGURE 7 | Suspected pox virus infection**.

#### **WEAKNESSES AND RECOMMENDATIONS**

Despite its positive impact, the present study identified various weaknesses to be addressed to ensure the long-term sustainability of the telemedicine system. We therefore make three main recommendations.

	- *Standardizing teledermatology clinical examination*. Fourteen percent of cases were reported by the expert as poor in quality or not good enough to make a diagnosis. If dermatological diagnosis is mainly clinical – based on visual inspection with no sophisticated investigations required – it is, however, not sufficient to make an accurate diagnosis. Our analysis confirms that non-specialist practitioners do not master dermatological language or basic knowledge that would allow them to send a complete and accurate medical report. A standardized teledermatology structured form could then both facilitate communication and improve diagnosis performed with the MSF teledermatology system (20) (**Figure 8**). This standardized form has been made available for all new dermatology cases since September 2014.
	- *Establish picture standards*. Fifteen percent of the attached pictures were considered by experts as of poor quality. Picture quality is crucial to reliable teledermatology diagnosis (21). Many studies have shown that when teledermatology relies on pictures of good quality it can deliver the same diagnosis that proper physical examination does in most cases (22). As with a face to face examination, pictures should represent the whole patient before providing close-ups of the primary lesion (23). As a standard, referrers should attach at least two pictures to all dermatology cases referred (one close-up, the other one an overview). Some classic general recommendations to standardize attached pictures are given in **Figure 9**.

3. *Provide more follow-up information*. One of the main concerns raised by experts in the survey was the lack of patient follow-up information. Only 10 cases (15%) had feedback data recorded. Most of the users agreed that follow-up was necessary (44% referrer majority response) or even mandatory (39% specialist majority response). This is necessary to keep the experts motivated and also to improve the quality of their answers and to allow them to learn from the referrer's feedback. By systematically recording patient outcomes, we can assess the real benefit, exploit statistics, and conduct proper scientific evaluation. Without feedback, the principle of requesting specialist advice is weakened, because no quality control is feasible. For facilitating the process, follow-up reports have been set up and are sent out automatically to the referrer.

## **CONCLUSION**

The present review shows that teledermatology is a growing part of a unique multilingual tele-expertise system supporting health professionals in the management of difficult clinical cases in the field. As shown in a previous larger survey (12), the majority of referrers (79%) reported that the advice received via the system improved their management of the patient. Nonetheless, user surveys and retrospective analysis suggest that the MSF teledermatology system can be improved. These improvements include providing information about system use, improving the quality of referrals and providing more follow-up information after teleconsultation. A future prospective evaluation could assess the impact of these recommendations.

## **REFERENCES**


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 05 August 2014; accepted: 27 October 2014; published online: 14 November 2014.*

*Citation: Delaigue S, Morand J-J, Olson D, Wootton R and Bonnardot L (2014) Teledermatology in low-resource settings: the MSF experience with a multilingual tele-expertise platform. Front. Public Health 2:233. doi: 10.3389/fpubh.2014.00233 This article was submitted to Public Health Education and Promotion, a section of the journal Frontiers in Public Health.*

*Copyright © 2014 Delaigue, Morand, Olson,Wootton and Bonnardot. This is an openaccess article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

## A retrospective analysis of pediatric cases handled by the MSF tele-expertise system

#### **Daniel Martinez Garcia<sup>1</sup>\*, Laurent Bonnardot 2,3, David Olson<sup>4</sup> , Harriet Roggeveen<sup>5</sup> , Jaap Karsten<sup>5</sup> , Peter Moons <sup>5</sup> , Myrto Schaefer <sup>1</sup> , Joanne Liu<sup>6</sup> and RichardWootton7,8**


#### **Edited by:**

Connie J. Evashwick, George Mason University, USA Karsten Lunze, Boston University, USA

#### **Reviewed by:**

Alain Dervaux, Centre Hospitalier Sainte-Anne, France Xiaoguang Ma, University of South Carolina, USA

#### **\*Correspondence:**

Daniel Martinez Garcia, Médecins Sans Frontières, Medical Department, C/Nou de la Rambla, 26. CP 08001 Barcelona, Spain e-mail: daniel.martinez@barcelona. msf.org

We conducted a retrospective analysis of all pediatric cases referred by Médecins Sans Frontières (MSF) field doctors via the MSF telemedicine system during a 4-year period from April 2010. A total of 467 pediatric cases were submitted, representing approximately 40% of all telemedicine cases. The median age of the patients was 4 years. The median response time (i.e., the interval between the case being submitted and the first response from a specialist) was 13 h (interquartile range 4–32 h). We selected a random sample of 12 pediatric cases in each of four age categories for detailed analysis by an experienced MSF pediatrician. In the 48 randomly selected cases, the mean rating for the quality of information provided by the referrer was 2.8 (on a scale from 1 = very poor to 5 = very good), and the mean rating for the appropriateness of the response was 3.3 (same scale). More than two-thirds of the responses were considered to be useful to the patient, and approximately three-quarters were considered to be useful to the medical team. The usefulness of the responses tended to be higher for the medical team than for the patient, and there was some evidence that usefulness to both groups was lower in newborns and adolescent patients. The telemedicine system allows the quality of the medical support given to medical teams in the field to be controlled objectively as there is a record of all cases and answers. Telemedicine has an important role in supporting the aims of medical humanitarian organizations such as MSF.

**Keywords: low income countries, limited resource settings, telemedicine, telehealth, pediatric, humanitarian, emergency medicine**

## **INTRODUCTION**

Pediatric cases represent the highest proportion of patients in Médecins Sans Frontières (MSF) programs. Providing rapid and useful support to the teams in the field is at the top of the organization's priorities. In MSF, most of the clinical work is conducted by general physicians, clinical officers, and nurses with very heterogeneous experience. This may include pediatric inpatient and outpatient care, malnutrition, or tropical medicine in limited resource settings.

Up to 55% of MSF medical activities are in conflict or unstable locations. Furthermore, almost all of MSF activities concentrate in remote or hard to reach areas. (1) This increases the challenge for medical staff and also delays time to consultation. Thus, many patients reach medical facilities with late presentations of diseases. Ensuring access to basic and vital medical services is at the heart of MSF objectives. Most MSF projects provide a range of generalist services in the field focusing on the most urgent needs of vulnerable populations. Some projects focus on a specific disease or medical condition that we consider neglected (e.g., HIV or TB); it is only in this type of project that access to specialized care will be possible routinely. In some settings, security risks are so high that it is very difficult to provide hands-on support to local medical teams. Telemedicine is one possible way in which access to specialist or more appropriate medical consultation might be improved.

In 2010, using a highly secure web-based messaging system, MSF began to pilot two telemedicine networks to support medical field workers. One was operated in French and one in English; a third Spanish network was brought into operation in 2012. In late 2013, the three telemedicine networks were combined into a single multilingual system, telemed.msf.org. A total of 1147 cases (both adult and pediatric) have been submitted through the system, and a survey showed that there was high satisfaction from the users (2).

In the integrated multilingual network, when a case is submitted a case-coordinator reviews the case to decide which specialist(s) will be the best to provide an answer. The coordinator then allocates the case. If an answer is not received within a certain time frame (maximum 24 h), the case is reallocated to another specialist, in order to ensure that the referrer in the field receives an answer in the shortest time possible.

Starting in October 2013, individual case follow-up (i.e., a progress report) was requested automatically from referrers, 21 days after each new case was submitted.

We have conducted a quantitative and qualitative analysis of the pediatric cases submitted through the system.

## **MATERIALS AND METHODS**

We conducted a descriptive retrospective analysis of all pediatric cases (age recorded under 18 years) referred by MSF field medical staff to the MSF telemedicine platform from April 2010 to March 2014, inclusive. This represented 467 cases out of 1147 cases in the system. The telemedicine system has a database from which authorized people can retrieve information from specific clinical cases, by selecting specific characteristics such as age. Ethics permission was not required, because patient consent had been obtained prior to submitting each case and the work was a retrospective chart review of anonymized data conducted by the organization's staff in accordance with its research policies.

## **CASE CHARACTERISTICS**

Demographic and other data were extracted from the database and stored in a spreadsheet for analysis of the case characteristics.

## **DETAILED REVIEW OF SAMPLE CASES**

In addition, because it was impractical to conduct a detailed review of the 467 pediatric cases, a random sample of approximately 10% of the cases was reviewed instead. The 48 cases were chosen randomly in four age groups, i.e., 12 pediatric cases were selected by stratified random sampling. The random selection was done using a randomization program according to the case number in the system. The age categories were: birth to 4 weeks, 1 month to 2 years, 2–10 years, and 10–18 years. The cases were assessed by an experienced MSF pediatrician who had both practised as a pediatric expert and as an MSF field referrer. He received the list of selected cases, obtained access to the electronic case record, and used a spreadsheet to summarize his findings as explained below. This reviewer was blinded to the process of selection of the 48 cases.

Three domains were assessed:

	- clear (easy to follow and implement),
	- accurate (medically in accordance with the best medical information available),

## **INDIVIDUAL FOLLOW-UP FROM THE REFERRER**

The progress reports based on closed-ended questions relative to the user's satisfaction and benefit were reviewed.

## **RESULTS**

## **CASE CHARACTERISTICS**

During the study period, a total of 467 pediatric cases were submitted by medical staff from MSF field sites. These pediatric cases comprised 41% of all telemedicine cases. Among the pediatric patients, there were 256 males and 201 females (in 10 cases, the sex of the patient was not recorded). The median age of the pediatric cases was 4 years (interquartile range 1–9 years). The number of patients in the four age categories was: 26 for 0–30 days, 155 for 1 month to 2 years, 193 for 2–10 years, and 93 for 10– 18 years (**Figure 1**). The cases were submitted from 28 countries

## **Table 1 | Countries of origin of cases**.


A small number of cases apparently from industrialized countries (shown in parentheses) were submitted via the MSF headquarters office responsible for a field hospital elsewhere.

(**Table 1**). Half of the cases were submitted from three countries: Central African Republic (23%), South Sudan (13%), and Ethiopia (11%).

Submission of the cases resulted in 761 queries (a case always results in at least one query being sent to a specialist; if there are requests for a subspecialist opinion, then a single case may result in several queries), a ratio of 1.6 queries per case. The median response time (i.e., the interval between the case being submitted and the first response from a specialist) was 13 h (interquartile range 4–32 h).

The queries covered a wide range of medical and surgical specialties (**Figure 2**). Among medical subspecialties, the three most common types of referral were for tropical diseases (36), dermatology (36), and neurology (9); among surgical subspecialties, the three most common types of referral were for ophthalmology (21), ENT (20), and orthopedics (18).

Over the 467 cases recorded, the majority (42%) were answered by pediatricians (**Figure 2**). This is not surprising as MSF pediatric advisers at headquarters are the first line responders for these cases and are often the focal point for centralizing advice from other experts and sub-specialists and thus provide a comprehensive answer to the field.

## **DETAILED REVIEW OF SAMPLE CASES**

In the 48 randomly selected cases, the mean rating for the quality of information provided by the referrer was 2.8 (**Table 2**, range 1–5), and the mean rating for the appropriateness of the response was 3.3 (range 1–5), implying an acceptable/good response given to the field. There was no evidence from the ratings in the subgroups that quality or appropriateness was substantially different across the different age groups of the patients, see **Table 2**.

Approximately two-thirds of the responses were considered to be useful to the patient, and approximately three-quarters were considered to be useful to the medical team. The usefulness of the responses tended to be higher for the medical team than for the patient, and there was some evidence that usefulness to both parties was lower for the newborns and the adolescent patients (**Figure 3**).

## **INDIVIDUAL FOLLOW-UP FROM THE REFERRER**

In the period October 2013 to March 2014 inclusive, 42 requests for follow-up were issued. A total of seven progress reports were provided (response rate of 17%). The responders were generally positive about the value of the teleconsultation (**Table 3**).

## **DISCUSSION**

We conducted a retrospective analysis of all pediatric cases referred by MSF field doctors via the MSF telemedicine system during a 4-year period.

## **REVIEW OF THE LITERATURE**

The majority of previous work on telepediatrics has been videobased, and has concerned high income countries. For example, much early work was done in Queensland (Australia) approximately 15 years ago (3, 4), and subsequently in the United States (US) (5). However, video links are expensive and the necessary bandwidth is not always available in low-resource settings. There has been little video-based work in low income countries: a pilot service in India (6) and the MSF Somalia project (7) are rare examples. The latter study reported that 346 cases (9% of the total) were referred for telemedicine and in 222 children (64%), "*a significant change was made to initial case management, while in 88 (25%), a life-threatening condition was detected that had been initially missed.*" Internet video has been used for surgical planning in some low and middle income countries (LMIC) prior to visits by US surgical teams (8, 9), but this did not specifically target the pediatric age group.

There has been a pediatric component to cases managed by store-and-forward telemedicine in other networks (e.g.,US Pacific, Swinfen), but no specific reporting of the experience in the literature. Some pediatric case reports have been provided. (10) Email has been used for pediatric orthopedics in Djibouti. (11) As far as we are aware, there are few other reports concerning store-and-forward telemedicine for pediatric work in low-resource settings.



<sup>a</sup>Scored from 1 = very poor to 5 = very good.

The columns "Useful to patient" and "Useful to medical team" contain some missing data (7 cases of the 48), where usefulness could not be determined.

#### **CHARACTERISTIC OF MSF PEDIATRIC FIELD WORK**

In the present study, there were a large number of X-rays. This was mainly due to the over-representation of HIV and TB projects using the system. Overall, more than 40% of cases submitted through the MSF telemedicine system involved patients under the age of 18 years. This is in line with the normal pyramid of ages in developing countries, where we would expect about 40–50% of the population to be less than 18 years of age.

The usefulness of the answers produced via the telemedicine system appeared to be lower for newborns and for adolescent patients (**Figure 3**). These two groups have features that require more knowledge and experience to deal with. Adolescents are a difficult group to reach and they often present with diseases at a more advanced stage. The newborn group is marked by congenital problems that are hard to diagnose without appropriate medical technologies in the field and which require highly specialized management. Most of the neonatal cases were advised by non-neonatologist specialists.

The MSF telemedicine system was initially designed to be used for complex cases (excluding life-threatening emergencies). However, it was often used for "non-complicated" cases for which protocols are available in MSF guidelines that could have been easily applied. (12) It is a fact that most MSF field work including pediatric clinical work is performed by general physicians, clinical officers, nurses, or midwives. The case content analysis revealed that in some cases the field medical teams did not even have a suspected diagnosis (working hypothesis). This might indicate the lack of familiarity of field workers with basic pediatric

principles and procedures, as well as a lack of use or appropriateness of existing tools such as guidelines. The role of the headquarters pediatric advisors is thus crucial to ensure that field teams do not overlook essential elements, physical findings, or specific information that would help them to manage cases by themselves. This also demonstrates the complexity of field management of pediatric cases and the growing need for having expert support for adapting protocols to the field environment.

## **STRENGTHS OF THE SYSTEM**

The review of the MSF tele-expertise system demonstrates some strengths. These include:


## **LIMITATIONS OF THE STUDY**

The present study only reports on clinical cases submitted through the telemedicine system. Thus, it does not provide a comprehensive picture of all MSF pediatric clinical cases. This is because a substantial proportion of cases requiring support from MSF pediatric advisors are sent from the field via other means, such as email, Skype, SMS, telephone, or even through social media platforms. For security and other reasons, MSF is trying to reduce the communication of clinical cases outside the telemedicine platform.

For the purpose of the present work, we only reviewed information available within the telemedicine system. As shown in **Table 1**, the use of the telemedicine system varied heavily, depending on the country and project acceptance and understanding of the system. The number of cases sent by a project was therefore a reflection of the acceptance of the telemedicine system in that particular country. Future comparisons will show the trends from each project and we will be able to assess acceptance and efficacy of the system.

A second limitation is that a retrospective analysis was carried out, with no control arm for a comparator. Moreover, the availability of only limited feedback about outcomes of cases may represent a source of bias. Although the preliminary feedback from referrers suggests that they find the system useful (**Table 3**), a systematic survey was not conducted. On the other hand, a much larger previous survey of users also found that their overall opinion was positive (2).

Finally, the qualitative assessment was done by a single pediatrician and therefore the results must be interpreted with caution. A future study would be strengthened if it used a panel of observers. Part of the validation of any future methodology would involve developing measures in which there was good agreement between and within observers.

## **PERSPECTIVES**

Despite the strengths of the telemedicine system, it also has weak points. This leads us to make the following recommendations:


## **Table 3 | Summary of the progress report data provided by referrers of pediatric cases**.


Free-text comments.

The outcome cannot be evaluated fully because the patient defaulted after the last necrosis was debrided, at least since the change of antibiotics there was no newly formed necrosis.

Patient follow-up has been lost.

Even if treatment options are limited at our level, advice on this difficult cases is very helpful to orient diagnosis and give patient proper advice.

The service is EXCELLENT, always well adapted to our environment, understanding of our limitations, and sometimes our lack of professionalism! Answers are always very rapid and extremely useful to the field and consequently the patients. We could not manage without them!.

Excellent service.

Telemedicine is appreciated a lot!.


## **FUTURE DEVELOPMENTS**

In the near future, with better reliability of new technology including mobile devices and a broader access to the Internet, we envisage that the telemedicine system will provide more direct support (e.g., at the bedside) to more field doctors. A telemedicine application for mobile devices would allow users to create their referral offline at the patient's bedside and then have it sent automatically as soon as an Internet connection was established. Real time telemedicine with live chat or real time video could also allow the telemedicine system to provide support for life-threatening emergencies, or for cases requiring very rapid decisions from the medical teams in the field. In order to be able to provide this new real time service, a pool of online experts would be required. A large telemedicine center, probably a virtual center, could be created to coordinate and respond to multiple cases being received.

## **CONCLUSION**

Given the significance of pediatric cases in the daily activities of MSF and the impossibility of having a trained pediatrician present at all field sites where children receive care, means that access to remote advice is important. A telemedicine system can greatly improve the level of medical care provided to patients and reduces the isolation of field doctors in their practice. Telemedicine is also valuable in insecure, unstable settings where the number of medical personnel needs to be minimized out of concern for staff safety.

Confidentiality and security of communication regarding patient information provided through telemedicine should lead healthcare organizations to consider using this as their sole method of communication with the field with regards to patient information. However, this will require easier access for field workers to the system, e.g., via mobile devices, and appropriate arrangements at headquarters level to manage the workload appropriately.

Medical humanitarian organizations such as MSF work to reduce the health gaps for the most vulnerable populations in the most difficult contexts. Telemedicine has an important role in supporting those aims.

## **REFERENCES**


Sans Frontières (2013). Available from: http://refbooks.msf.org/msf\_docs/en/ clinical\_guide/cg\_en.pdf

13. Jakowenko J, Wootton R. An analysis of the images attached to referral messages in an email-based telemedicine system for developing countries. *J Telemed Telecare* (2006) **12**(Suppl 3):49–53. doi:10.1258/135763306779380066

**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 18 September 2014; paper pending published: 24 October 2014; accepted: 17 November 2014; published online: 08 December 2014.*

*Citation: Martinez Garcia D, Bonnardot L, Olson D, Roggeveen H, Karsten J, Moons P, Schaefer M, Liu J and Wootton R (2014) A retrospective analysis of pediatric cases handled by the MSF tele-expertise system. Front. Public Health 2:266. doi: 10.3389/fpubh.2014.00266*

*This article was submitted to Public Health Education and Promotion, a section of the journal Frontiers in Public Health.*

*Copyright © 2014 Martinez Garcia, Bonnardot , Olson, Roggeveen, Karsten, Moons, Schaefer, Liu and Wootton. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

## Assessing the quality of teleconsultations in a store-and-forward telemedicine network

## **RichardWootton1,2\*, Joanne Liu<sup>3</sup> and Laurent Bonnardot 4,5**

<sup>1</sup> Norwegian Centre for Integrated Care and Telemedicine, University Hospital of North Norway, Tromsø, Norway

<sup>2</sup> Faculty of Health Sciences, University of Tromsø, Tromsø, Norway

<sup>3</sup> Médecins Sans Frontières International, Geneva, Switzerland

<sup>4</sup> Department of Medical Ethics and Legal Medicine, Paris Descartes University, Paris, France

<sup>5</sup> Fondation Médecins Sans Frontières, Paris, France

## **Edited by:**

Michal Grivna, United Arab Emirates University, United Arab Emirates

#### **Reviewed by:**

Donghua Tao, Saint Louis University, USA

Janice Elisabeth Frates, California State University Long Beach, USA

#### **\*Correspondence:**

Richard Wootton, Norwegian Centre for Integrated Care and Telemedicine, University Hospital of North Norway, PO Box 6060, Tromsø 9038, Norway e-mail: r\_wootton@pobox.com

Store-and-forward telemedicine in resource-limited settings is becoming a relatively mature activity. However, there are few published reports about quality measurement in telemedicine, except in image-based specialties, and they mainly relate to high- and middle-income countries. In 2010, Médecins Sans Frontières (MSF) began to use a store-and-forward telemedicine network to assist its field staff in obtaining specialist advice. To date, more than 1000 cases have been managed with the support of telemedicine, from a total of 40 different countries. We propose a method for assessing the overall quality of the teleconsultations provided in a store-and-forward telemedicine network. The assessment is performed at regular intervals by a panel of observers, who – independently – respond to a questionnaire relating to a randomly chosen past case.The answers to the questionnaire allow two different dimensions of quality to be assessed: the quality of the process itself and the outcome, defined as the value of the response to three of the four parties concerned, i.e., the patient, the referring doctor, and the organization. It is not practicable to estimate the value to society by this technique. The feasibility of the method was demonstrated by using it in the MSF telemedicine network, where process quality scores, and user-value scores, appeared to be stable over a 9-month trial period. This was confirmed by plotting the cusum of a portmanteau statistic (the sum of the four scores) over the study period. The proposed quality-assessment method appears feasible in practice, and will form one element of a quality assurance program for MSF's telemedicine network in future. The method is a generally applicable one, which can be used in many forms of medical interaction.

**Keywords: telemedicine, telehealth, quality assurance, process control, LMICs**

## **INTRODUCTION**

Médecins Sans Frontières (MSF) is a non-governmental humanitarian medical organization that responds to emergency situations and provides medical assistance to those in need. MSF teams provide medical emergency aid in difficult settings around the world, and staff often have to diagnose and treat patients with limited resources (1). In 2010, MSF began to use a store-and-forward telemedicine network to assist its field staff in obtaining specialist advice (2). To date, more than 1000 cases have been managed with the support of telemedicine, from a total of 40 different countries.

In a store-and-forward telemedicine network of this type, doctors in the field refer cases electronically to obtain a second opinion about diagnosis or management. Incoming cases are reviewed by a case coordinator and assigned for reply to one or more appropriate experts. The network therefore operates in a similar way to a bulletin board, with messages being posted by its users. Although formal evidence for the clinical effectiveness of the telemedicine advice obtained through networks of this kind is rather scarce (3, 4), they are known to provide a useful service to referring doctors, and several networks have operated for periods of more than a decade.

## **QUALITY PROBLEM**

As store-and-forward telemedicine in resource-limited settings is becoming a relatively mature activity, there is a concomitant requirement to implement quality assurance/improvement activities. Indeed, it may be considered unethical not to do so. However, there are few published reports about quality measurement in telemedicine, except in networks concerned with radiology (5), ophthalmology (6), or histopathology (7), many of which are retrospective studies. These reports concern image-based activities, which perhaps lend themselves more readily to quality measurement. The situation in teleconsulting is more complex, being inherently multi-specialty in nature and one where there is often limited knowledge of outcomes. Attempting to measure quality in such a context is more like attempting to measure overall quality in a multi-clinic outpatient department. As far as we are aware, there have been no previous studies of prospective quality measurement in general teleconsulting work in low income countries.

## **OBJECTIVES**

The primary research question was whether a method could be developed for quality measurement in general teleconsulting work in low income countries. The aim of the present work, therefore, was to develop a method for assessing the quality of the teleconsultations being conducted in the MSF telemedicine network, and then to examine its feasibility for routine adoption.

## **MATERIALS AND METHODS**

The present study required the development of a method to assess quality and then a demonstration of its feasibility in practice. The work was performed in two stages:


Ethics permission was not required, because patient consent to access the data had been obtained and the work was a retrospective chart review conducted by the organization's staff in accordance with its research policies.

## **ASSESSMENT OF QUALITY Development of the quality tool**

A questionnaire was developed by a consensus between three experienced telemedicine practitioners. It was based on accepted tools used in previous studies (8, 9). The final questionnaire was evaluated and approved by an independent evaluator. The final questionnaire consisted of 17 questions. These concerned the information provided by the referring doctor, the way that the referral was handled in the telemedicine network, the response(s) received from the specialist(s) consulted, and the likely value to the patient, the doctor, and the organization.

## **Definition of quality**

We defined quality in terms of two of the three dimensions of the Donabedian model: process and outcome. (The structural dimension is not usually relevant in a telemedicine network of the sort under discussion.) Thus in assessing the quality of a given teleconsultation, there are two principal questions:


These questions address separate dimensions of quality, both of concern to network operators. That is, the process for producing a response might be satisfactory, but the response itself could be useless. Or the process could be unsatisfactory, but the response might still be useful.

Both aspects of quality can best be judged by using a panel of assessors. This is because any evaluation will involve subjective judgments, so a panel of observers is more likely to produce an accurate estimate than a single observer. However, it is not feasible to evaluate the quality of every single teleconsultation conducted in the network, so there must be a sampling process by which a case is selected (randomly) for assessment at regular intervals. This leads to a quality-assessment scheme whose main features are summarized in **Table 1**.

## **Quality of process**

The quality of the teleconsultation process (*q*p) can be assessed by the panel members, who can make a judgment about various relevant matters. For example, they can judge whether the referrer provided sufficient information, whether the case was sent promptly to an appropriate expert, whether an answer was obtained sufficiently quickly to be useful and so on. There are 10 questions listed in **Table 2** which are relevant to the quality of the process. The scoring system is described in Appendix.

## **Value of response**

The value of the response can be assessed in a similar way by the panel members. There are four domains of interest:


The first three values can be assessed by staff with suitable telemedicine experience. However, assessing the value to society is much more difficult. The value to society of telemedicine will be partly determined by the health care system in the country concerned (mainly,the country where the patient is located),including the degree to which telemedicine has been properly integrated into the chain of health care there. Assessing the value to society as a whole is therefore difficult to do on the basis of a single telemedicine case, and is ignored in what follows. It is worth noting that in a humanitarian context (or a not-for-profit operation), the value to society will be closely aligned with the value to the organization.

Direct measurement of value is not straightforward. In health economics, it is usual to measure the cost-effectiveness of the technique in question and to make a comparison (e.g., with usual practice) to obtain evidence that it does not represent a waste of resources. However, in the context of telemedicine in resourcelimited settings, this is not easy to do. First, the costs are distorted, because many staff are volunteers and there may also be donor support, which can be hard to quantify. Second, the clinical effect of telemedicine may be difficult to document, as patients are commonly lost to follow up after their initial encounter.

How else can the "value" of a teleconsultation episode be measured? That is, what is the value to the interested parties? Panel members can form a judgment about whether the telemedicine response clarified the diagnosis, whether the eventual clinical outcome would be beneficial for the patient and so on. There are nine questions listed in **Table 2** which are relevant to the value of the response in the domains of interest. The scoring system is described in Appendix.

## **Table 1 | Main features of the quality-assessment scheme**.


## **DEMONSTRATION OF FEASIBILITY**

To demonstrate the feasibility of the proposed approach, a panel of 12 experts was invited to answer the 17 questions about randomly selected telemedicine cases, see **Table 2**. Cases were chosen at random for a 9-month period. The process was as follows:


## **Process stability**

A control chart was used to examine the stability of the monthly quality scores. Control charts can be plotted for each of the four quality indices, but for simplicity, a grand quality score (GQS) for each case was calculated from the panel's quality and value scores as

$$G = \begin{array}{c} Q\_{\mathbb{P}} \ + \ V\_{\mathbb{P}} \ + \ V\_{\mathbb{r}} \ + \ V\_{\mathbb{o}} \end{array}$$

That is, the GQS represents an equi-weighted summation of the four constituent indices. The GQS was transformed to lie in the range 0–10 (0 = worst, 10 = best).

The cusum chart is a well-established and powerful method for identifying changes in a process average. The chart plots the cumulative difference between the recorded values and a target value, which is often chosen to be the process average. The GQS values

were plotted as a cusum, using the grand mean as the reference value.

Note that there are two important assumptions underlying the use of control charts: the measurement that is used to monitor the process is distributed according to a normal distribution; it was not necessary to transform the data in the present case. Also, the measurements are assumed to be independent of each other.

## **RESULTS**

The panel assessed randomly selected cases starting in July 2013. At least four responses were received for each case. The median panel score for process quality was 8.0 (IQR 7.3, 8.7) across the nine cases. The lowest score awarded for process quality by an individual panel member in any case was 4.7 and the highest was 9.0. The median values in each case are shown in **Figure 1**. There was good agreement between panel members about process quality, i.e., relatively small IQRs for each case.

The median panel score for value to the patient was 8.9 (IQR 7.8, 8.9). The lowest score awarded for value to the patient was 3.3 and the highest was 10. The median values in each case are shown in **Figure 2**. The agreement between panel members was less good than for process quality.

The median panel score for value to the doctor was 9.1 (IQR 8.6, 9.5). The lowest score awarded for value to the doctor was 5.7 and the highest was 10.0. The median values in each case are shown in **Figure 3**. The agreement between panel members was better than for value to the patient.

The median panel score for value to the organization was 8.9 (IQR 7.2, 10.0). The lowest score awarded for value to the organization was 5.6 and the highest was 10.0. The median values in each case are shown in **Figure 4**. The agreement between panel members was less good than for value to the doctor.

The median panel GQS was 8.6 (IQR 7.6, 9.2). The lowest individual GQS was 6.1 and the highest was 9.8. The median values in each case are shown in **Figure 5**. The cusum is shown in **Figure 6**. There was no evidence that the process was out of control, i.e., with steadily increasing or steadily decreasing values. In fact, over the epoch studied, the cusum was essentially zero at the end, while deviations no larger than ±12% occurred over the study period.

## **DISCUSSION**

We have developed a quality-assessment scheme for a store-andforward telemedicine network and demonstrated its feasibility in

## **Table 2 | Quality-assessment questions**.


a real-life clinical setting. There appear to be no previous reports of similar work.

## **RELATION TO OTHER EVIDENCE**

Previous work on assessment of quality in telemedicine networks has often focused on user satisfaction [e.g., Ref. (10)], which is a related, but different, concept. Most previous quality studies have been retrospective reviews, such as that conducted by Mahnke et al. (11). There have been few attempts to measure the value to the clinician, although Chan et al. investigated this in a real-time teleconsultation network in a high-income country (12).

## **METHODOLOGICAL ISSUES**

The proposed method was trialed in a real-life telemedicine network, where it was shown to be feasible and appeared to produce useful results. It thus appears suitable for routine adoption. Implicit in the methodology are a number of design decisions.

## **Questionnaire**

The size of the questionnaire is likely to influence the number of responses from the panel. The right balance has to be struck between asking too few questions and too many. On one hand, the more questions that are asked, the better the situation can be

assessed; but on the other hand, too many questions will discourage the observers from responding, which will make the system less sustainable. In practice, 10–20 questions seems to be a reasonable number.

## **Monitoring and stability**

Which index (process quality and the three value domains) is most appropriate for long-term monitoring, in order to measure network performance? Are all four indices of equal importance, or should some be more heavily weighted than others? Should they be monitored collectively, rather than individually? This requires further work.

**FIGURE 3 | Median scores for value to referrer (0** = **worst; 10** = **best)**. The error bars indicate the 25th and 75th percentiles.

## **Sampling**

How often should cases be sampled and monitoring be performed? On one hand, more frequent sampling will allow closer performance monitoring; on the other hand, it is likely to lead to "observer fatigue." In practice, we suggest that random sampling of one case per month is about right.

## **Size of panel**

How many panel members should give an opinion? The more members there are, the more likely there is to be disagreement between them; on the other hand, the more there are, the better

the estimate of the underlying value. In practice, we suggest that 5–10 panel members are about right.

## **QUALITY ASSURANCE**

Routine measurement of quality on randomly selected cases is only one part of the whole evaluation process and will form one element of an overall quality assurance program for the telemedicine network concerned. Other elements may include obtaining other points of view and follow up reports to assess long-term outcomes concerning the cases and the benefits of the expertise.

## **LIMITATIONS**

The present work has certain limitations. For example, before it could be used routinely, the quality-assessment methodology would require validation. However, it is difficult to validate the proposed indices independently, especially in the context of a telemedicine network operated by a humanitarian organization. Ideally, they should be evidence-based, and of demonstrated validity and reliability (13). Further work is required to find out whether this is possible, since the practical problem of the lack of an obvious gold standard needs to be overcome. Validation may therefore need to rest on psychometric methods (14).

Industrial process control is normally done using an absolute standard as the reference. In the present work, a relative reference value was employed. That is, it represents an assessment of relative quality, which pragmatically, is probably better than no assessment at all. Again, further work is required to find out whether absolute reference standards can be developed.

Finally, the quality of this evaluation relies on the information available for assessing the case. Sampling a case at a particular time may be problematic if there is insufficient feedback on follow up. It also relies on the expertise and experience of the assessor panel. The panel members must be selected carefully and it is important that they have no conflict of interest. This is why it may be better

to use independent volunteers, rather than senior staff from the organization running the network.

## **INTERPRETATION**

The present method provides estimates of the value to the main parties concerned in a teleconsultation, together with an estimate of the quality of the teleconsultation process itself. This is important information for those responsible for the operation of the network. To the best of our knowledge, there has been no information published previously about the quality of general teleconsultations in a store-and-forward network. Yet, if telemedicine is considered sufficiently mature that it can enter routine service, there is an ethical imperative to ensure that it is employed in a cost-effective manner. The method described here provides an instrument for monitoring quality and will form part of the toolset used by the operators of the MSF telemedicine network in future.

Once a method for assessing quality is available, application of industrial process control methodology allows the stability of the network to be monitored. Again, this is important if network operators are to be reassured that quality is not in slow decline. The information may also be valuable in improving the performance of healthcare staff in low-resource settings, which is known to be a difficult problem (15).

The techniques presented in this paper are of wide application. They could potentially be used in non-telemedicine consultations (i.e., conventional, face-to-face consulting), and in industrialized countries as well as resource-limited settings.

## **CONCLUSION**

A method for assessing the quality of the teleconsultations in a store-and-forward telemedicine network is proposed. It provides estimates of the quality of the process and the value of the consultation to the main parties involved. A trial of the method showed that it was feasible and that the process in the network studied was stable. The method appears to give useful results. It seems desirable to implement it in other telemedicine projects where it can contribute to the evaluation of practice, something that is necessary in all medical services provided.

## **ACKNOWLEDGMENTS**

We are grateful to Will Wu for software support.

## **REFERENCES**


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 12 May 2014; accepted: 01 July 2014; published online: 16 July 2014. Citation: Wootton R, Liu J and Bonnardot L (2014) Assessing the quality of teleconsultations in a store-and-forward telemedicine network. Front. Public Health 2:82. doi: 10.3389/fpubh.2014.00082*

*This article was submitted to Public Health Education and Promotion, a section of the journal Frontiers in Public Health.*

*Copyright © 2014 Wootton, Liu and Bonnardot. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited andthatthe original publication inthis journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

## **APPENDIX**

## **SCORING OF QUESTIONS**

The questions set out in **Table 2** are presented as multiple choices for simplicity. Scoring for most questions – those in which an affirmative response indicated satisfaction with the item being considered – was No = 1, Perhaps = 2, and Yes = 3. However, some questions required reversed scoring, where an affirmative indicated dissatisfaction: Yes = 1, Perhaps = 2, and No = 3.

Do not know responses were coded as zeroes, i.e., they were treated in the same way as missing values. That is, no distinction was made in the present work between missing values and do not know responses. These may represent cases where assessors experienced particular difficulty in forming a judgment.

## **Individual scores**

Scores were first calculated for each individual panel member, as follows. The process quality score from a panel member who had assessed a particular case was calculated as follows:

$$q\_{\mathbb{P}} = s\_1 + s\_2 + s\_4 + s\_5 + s\_6 + s\_7 + s\_8 + s\_9 + s\_{10} + s\_{15}$$

where the response elements, *s*1, *s*2, … refer to the responses in **Table 2**, reverse-scored where appropriate.

The value scores from a given panel member were calculated as follows:

$$\begin{aligned} \nu\_{\rm p} &= s\_{11} + s\_{12} + s\_{13} \\ \nu\_{\rm r} &= s\_{7} + s\_{8} + s\_{9} + s\_{10} + s\_{11} + s\_{12} + s\_{14} \\ \nu\_{\rm o} &= s\_{13} + s\_{14} + s\_{16} \end{aligned}$$

In the above calculations, the response elements for a given panel member were aggregated by simple summation. That is, the constituent responses for the questions being used in a particular score were equi-weighted.

For convenience, the values *q*p, *v*p, *v*<sup>r</sup> , and *v*<sup>o</sup> were transformed to lie in the range 0–10 (0 = worst, 10 = best).

## **Panel scores**

The individual panel member scores were aggregated to produce a panel mean, which represents the best estimate of the underlying true value pertaining to the case in question. In the absence of a compelling reason to use a more complex scheme, the individual scores were equally weighted, i.e., this amounts to placing similar value on the judgment of all members of the panel. For example, the panel's best estimate of process quality was

$$Q\_{\mathbb{P}} = (q\_1 + q\_2 + \dots + q\_{\mathbb{H}})/n$$

where *Q*<sup>p</sup> is the panel process quality score, and *q*1, *q*<sup>2</sup> … are the individual scores for process quality from the *n* panel members.

METHODS ARTICLE published: 28 October 2014 doi: 10.3389/fpubh.2014.00211

## Assessing the quality of teleconsultations in a store-and-forward telemedicine network – long-term monitoring taking into account differences between cases

## **RichardWootton1,2\*, Joanne Liu<sup>3</sup> and Laurent Bonnardot 4,5**

<sup>1</sup> Norwegian Centre for Integrated Care and Telemedicine, University Hospital of North Norway, Tromsø, Norway

<sup>2</sup> Faculty of Health Sciences, University of Tromsø, Tromsø, Norway

<sup>3</sup> Médecins Sans Frontières International, Geneva, Switzerland

<sup>4</sup> Department of Medical Ethics and Legal Medicine, EA 4569, Paris Descartes University, Paris, France

<sup>5</sup> Fondation Médecins Sans Frontières, Paris, France

#### **Edited by:**

Connie J. Evashwick, Harvard School of Public Health, USA

#### **Reviewed by:**

Sheilla K. Pinjani, Aga Khan University, Pakistan Allen C. Meadors, University of North

Carolina at Pembroke, USA

#### **\*Correspondence:**

Richard Wootton, Norwegian Centre for Integrated Care and Telemedicine, University Hospital of North Norway, PO Box 6060, Tromsø 9038, Norway e-mail: r\_wootton@pobox.com

We have previously proposed a method for assessing the quality of individual teleconsultation cases; this paper proposes an additional step to allow the long-term monitoring of quality. The basic scenario is a teleconsultation system (aka an e-referral system or a teleexpertise system) where the referrer posts a question about a clinical case, the question is relayed to an appropriate expert, and the chosen expert provides an answer. The people running this system want assurances that it is stable, i.e., they want routine quality assurance information about the "output" from the "process." This requires two things. It needs a method of assessing the quality of individual patient consultations. And it needs a method for taking into account differences between patients, so that these quality assessments can be compared longitudinally. Building on the previously proposed methodology, the present paper proposes two techniques for measuring the difficulty posed by a particular teleconsultation.The first is an indirect method, similar to a willingness to pay economic estimation. The second is a direct method. Using these two methods with real data from a telemedicine network showed that the first method was feasible, but did not produce useful results in a pilot trial. The second method, while more laborious, was also feasible and did produce useful results. Thus, when output quality is measured, an allowance can be made for the characteristics of the case submitted. This means that fluctuations in output quality can be attributed to variations in the process (network) or to variations in the raw materials (queries submitted to the network). Long-term quality assurance should assist those providing telemedicine services in low-resource settings to ensure that the services are operated effectively and efficiently, despite the constraints and complexities of the environment.

**Keywords: telemedicine, telehealth, quality assurance, quality control, LMICs**

## **INTRODUCTION**

Telemedicine has been used for many years to support doctors working in low-resource settings. Sometimes real-time telemedicine is used, for example, video links between a doctor in the field and a specialist, but more commonly store-and-forward telemedicine is employed, because it is cheaper and easier to organize. Médecins Sans Frontières (MSF), a non-governmental humanitarian medical organization, has used both approaches (1–3). The store-and-forward telemedicine network, which it currently operates can be viewed as a logical development of its work, where doctors working with scarce resources in remote settings can obtain specialist medical advice for specific patients.

As telemedicine matures and becomes adopted as a routine method of healthcare delivery, there is an obligation to implement quality assurance/improvement activities. All provider organizations need to demonstrate that they are providing high-quality care via validated and controlled tools.

## **OPERATION OF A TELEMEDICINE NETWORK**

A store-and-forward telemedicine network of the type under discussion provides "tele-expertise" to doctors in the field. These field users can submit clinical queries to the network, and based on some internal mechanism (not relevant here), the query is sent to an appropriate expert for reply. In other words, the telemedicine network can be regarded as a "black box" (4), which accepts an input, carries out some action, and produces an output. That is, a clinical query is put into the black box, it is processed in some way, and an expert answer comes out. In the longer term, this can be viewed as a production process, similar to the manufacture of goods in factory: raw materials arrive, they are processed, and the resulting goods represent the output.

## **STATEMENT OF THE PROBLEM**

If a telemedicine network is viewed as a black box, then industrial methods for controlling the process become relevant. In industrial production processes, it is usually desirable to measure the quality of the output and ensure that this meets some target value. To do this, the output from a production run is sampled intermittently and judged against a suitable standard. For example, the output from a factory bottling wine might be judged by weighing the bottles to confirm that they had been filled satisfactorily. Let us suppose that the target weight for the contents of the bottles is 700 g. A sample bottle can be weighed when empty and again after it has been filled, allowing the weight of the contents to be determined accurately. To carry out quality control, bottles will be sampled regularly and the content weights will be plotted on a process control chart. The filling process will be considered satisfactory if the average content weight is sufficiently close to the target and there are no indications that the average weight is drifting either up or down. Conventional process control therefore depends on a method for measuring the output achieved and a comparison with a target (the desired output).

Now consider the quality of teleconsultations, selected from the "output" of a telemedicine network. Again, the process operators (i.e., the people responsible for running the network) may wish to know that the process is stable. That is, they want confirmation that the quality of the teleconsultations is satisfactory and that the average quality is not declining. (They may not object if the average quality is increasing, of course.)

In conventional process control, the output from the telemedicine network would be measured, and compared with a predetermined target value. We have previously described a method for measuring output (5), but it is not straightforward to define a target value for a telemedicine network. A quantitative description of the quality of an ideal teleconsultation is not possible in the current state of our knowledge. Instead, for process control purposes, we propose the measurement of the quality of the *input* to the process, so that observed fluctuations in output quality can be attributed to variations in the process (network) or to variations in the raw materials (queries submitted to the network).

*Measuring the output*. Our previous paper (5) sets out a method by which output quality can be assessed. A panel of observers makes a judgment about whether various aspects of the teleconsultation are considered satisfactory or not. These assessments are made without an explicit quality standard – they are actually based on the judgment made by each observer, which are aggregated to form a panel view. Thus, the assessment relates to the patient in question, but there is no way of accounting for the fact that patients differ. In other words, these are assessments of quality achieved in individual cases, but in the absence of a target (the desired quality), it is very difficult to monitor any long-term drift in quality.

*Measuring the input*. This requires a method for assessing the submitted queries and the resources available to the network for answering them. In real life, the "production setting" differs each time a query is submitted because patients are different from one another. Sometimes, the telemedicine question may be straightforward, e.g.,"*here is a chest X-ray; doesthis show TB?*"Sometimes, the telemedicine question may be very complex, e.g.,"*here is some clinical history; we don't know what the diagnosis is and we don't*

*know how to manage the patient; can you help?*" The production setting can also be made more difficult by the non-availability of specific experts required for certain cases. Thus, if the difficulty of the cases is not taken into account, it is not possible to assess output quality longitudinally.

Process control in a telemedicine network therefore requires two things. It needs a method of assessing the quality of the individual patient consultations, which are produced. And it needs a method for taking into account differences between cases, so that these quality assessments can be compared. To allow for the differences between patients, we need a method of measuring the "difficulty" of the question being posed to the teleconsultation network.

An analogy is a process, which produces a food – say, airline meals – from a raw material. The raw materials (ingredients) vary from batch to batch, but the process operators require the product to be as consistent as possible. So in some batches, much more skill is required (i.e., if the case in question is "difficult"). In the commercial kitchen example, this might mean preparing the food at a different temperature and/or for a different time. There may be instances where such a poor batch of ingredients is supplied that the quality of the product suffers. Quality monitoring would then show that this batch was of lower quality, and would also reveal the reason why: the "case" was extremely difficult because of substandard raw material. In other instances, a poor batch of ingredients might be supplied, yet the skill of the production operatives (chefs) might ensure that the output quality was normal.

Thus, the problem addressed in the present work is the development of a method that can be used by the people responsible for running a telemedicine network to monitor its operation with the aim of determining whether the process is stable and whether the quality of the teleconsultations is being maintained. This requires a method for taking into account differences between cases, so that these quality assessments can be compared. As far as we are aware, there has been no previous work on this subject.

## **DIFFICULTY OF THE TELECONSULTATION CASE**

The difficulty of a submitted case will be partly dependent on the clinical complexity of the patient. (Only partly, because we could have a complex question about a simple clinical problem or vice versa). In fact, the difficulty of the case depends on four main factors:


That is, from the point of view of the telemedicine network that receives a new case, it may be difficult to provide an answer because the problem is badly described, because the patient has a very complex illness, because the network does not have the right expert available to respond, or because the case is being managed in a remote hospital where treatment options are likely to be limited. Some or all of these difficulties may be present in any given case. Furthermore, each of these factors depends on various sub-factors:


The situation is summarized in **Table 1**. Thus, if we measure the outputs from a telemedicine network and find that output quality is declining, we want to be able to distinguish between a problem with the production process itself and a problem with the raw material (i.e., a more complex patient or a poorly described question from the referrer).

#### **OBJECTIVE**

The objective of the present work was to develop a method for determining the difficulty of a case being submitted for teleconsultation, able to take into account the differences between patients.

## **METHODS**

We propose two methods for determining the difficulty of the case submitted to a teleconsultation network. The first is indirect, and the second is direct. The feasibility of each method was trialed using data from an operational telemedicine network. Ethics permission was not required, because patient consent had been obtained prior to submitting each case and the work concerned the retrospective review of anonymized data conducted by the organization's staff in accordance with its research policies.

## **FIRST METHOD – INDIRECT ASSESSMENT**

## **Background – willingness to pay**

In health economics, an established technique for estimating the value of a product is to find out people's willingness to pay (WTP) for it. Technically, WTP is the maximum amount that a person is willing to sacrifice to procure a good or to avoid something



undesirable. This is usually established by surveying a group of consumers who are asked questions such as,"Would you purchase this product if it were offered at a price of X?" If this price differs between the consumers surveyed, then it is possible to make a good estimate of the sample's collective WTP a particular price.

Willingness to pay surveys have been used in medicine generally and in telemedicine specifically. For example, in one of the earliest telemedicine studies, Tsuji et al. (8) surveyed users of a home telemonitoring system in Japan; the best estimate of the WTP was U4519 (approximately US\$37) per user per month (8). Bergmo and Wangberg surveyed patients in a Norwegian general practice to investigate their WTP for teleconsultations. Approximately half of the patients were willing to pay for electronic contact with their GP (9). Bradford et al. investigated the willingness of patients with chronic heart failure to pay for access to medical care via telemedicine, as an alternative to traveling to the physician's office. They found that 55% of the patients surveyed would be willing to pay \$20 to access telemedicine instead of traveling to the physician's office, for at least some of their care (10).

## **Estimation of case difficulty**

We have previously proposed a method for assessing the quality of a teleconsultation, which requires a panel of observers to answer questions about a selected case. The method provides indices (scores) relating to different aspects of quality (5). The present proposal extends this methodology to take account of the difficulty posed by an individual case. This is estimated by a consensus among those reviewing the case, as follows.

Suppose four panel members review a case, answer the value questions independently, but are not told what the overall value (score) of their responses is. Then they are asked a final question: "Considering the teleconsultation as a whole, do you think the quality (value) was sufficiently good in the circumstances? In other words, quality can always be made higher, but was it good enough?"

That is, the final question takes into account the specificity of the environment and its variability. Their individual answers to this question might be:

$$\text{Y, Y, N, N}$$

If the corresponding quality scores (i.e., each member's assessment of the value achieved, on a Likert scale from 0 to 10) are computed, these might turn out to be:

$$9.1, \ 8.5, \ 6.9, \ 7.5$$

From the first two responses, we know that the score of 9.1 was considered high enough (by panel member 1), but that a score of 8.5 was also considered high enough (by member 2). That is, 8.5 represents the upper bound on the quality required.

From the other answers, we know that 6.9 was not considered high enough (by member 3) and that 7.5 was not considered high enough either (by member 4). That is, the lower bound lies *just above* 7.5. In the scoring system under discussion, a precision of more than 1% would not be meaningful. Thus, a lower bound lying just above 7.5 can be taken as a value of 7.6.

Therefore, in this example, the value can be estimated to lie in the range 7.6–8.5. This represents a consensus view about the quality of the teleconsultation, taking into account the circumstances of the case, such as whether the clinical question was very complex.

In establishing the consensus view of the panel, there are three possible sets of answers:


These three scenarios are depicted in **Figure 1**.

## **Feasibility**

To examine the feasibility of this approach, we used it prospectively on cases from the MSF telemedicine network. A panel of observers rated seven cases, which were being assessed routinely for quality assurance purposes.

## **Results**

The responses of the panel are summarized in **Table 2**. The second column contains the answers of each panel member to the question "Considering the teleconsultation as a whole, do you think the quality (value) was sufficiently good in the circumstances? In other words, quality can always be made higher, but was it good enough?" The third column contains, the overall quality score assigned by that panel member, but not revealed to them at the time they answered the question. The fourth column represents, the bound deduced from the panel member's response, and the

fifth column is the estimated value based on the responses from the whole panel.

Note that the panel estimate was considerably higher in case 914 than in cases 1201 and 1221. This suggests that the latter cases are more "difficult." Case 914 concerned a request for interpretation of chest X-ray images; this was a relatively straightforward query for the network to handle. Case 1201 was a patient with penile wounds, and case 1221 concerned loss of vision in a patient with multiple drug-resistant TB; both cases can be considered as fairly complicated queries. However, in four of the seven cases, the panel's estimate was only determined as an upper boundary (e.g., <6.2) rather than a specific value.

## **SECOND METHOD – DIRECT ASSESSMENT**

An alternative method of assessing the difficulty of the question in a teleconsultation network is direct estimation, by having an expert panel rate the difficulty of each case explicitly. That is, suitably qualified observers would independently assess teleconsultation cases by answering the 11 questions about each case shown in **Table 3**. The scores are then combined by simple summation to produce a rating of difficulty.

## **Feasibility**

Three observers (experienced telemedicine case-coordinators) independently rated 10 telemedicine cases selected randomly from previous cases in the MSF telemedicine network.

## **Results**

The mean score for difficulty (0 = no difficulty to 33 = extreme difficulty) ranged from 19 (case 1019) to 24 (case 1082), see **Figure 2**. That is, the difficulty of case 1082 was considered by

**Table 2 | Indirect assessment of case difficulty**.


the panel to be much higher than that of case 1019. Case 1082 concerned a child of 11 months admitted 3 days previously with an unclear history; this could certainly be considered to be a complicated query for the network to handle. Case 1019 concerned the management of a baby aged 5 weeks with an established diagnosis of osteomyelitis; this was a relatively straightforward query for the network to handle.

## **DISCUSSION**

There are few published reports about quality measurement in telemedicine. Most have been retrospective studies, and concern specific application areas such as radiology (11), ophthalmology (12), or histopathology (13). That is, these reports concern imagebased activities, which perhaps lend themselves more readily to quality measurement. In comparison, the situation in teleconsulting is more complex, being inherently multi-specialty in nature and one where there is often limited knowledge of outcomes. Attempting to measure quality in such a context is more like attempting to measure the quality of the consultations taking place in a multi-clinic outpatient department. As far as we are aware, there have been no previous studies of prospective quality measurement in general teleconsulting work in low-income countries. Furthermore, we are unaware of work concerning the differences between cases in a teleconsulting network.

The present work sets out what is required for long-term monitoring of quality in a teleconsulting network. In conventional process control, the output from the telemedicine network would be measured, and compared with a target value. Since it is not straightforward to define the latter, we propose the assessment of the input to the process instead. When each quality measurement of the output is made, an allowance can be made for the characteristics of the case submitted. This means that fluctuations in output quality can be attributed to variations in the process (network) or to variations in the raw materials (queries submitted to the network).

Two methods of estimating the degree of difficulty posed by cases submitted to a telemedicine network have been trialed. The first, an indirect method, is easier to use in practice, but a pilot study shows that it produces results of limited value. The second method, the direct estimation of case difficulty, is more demanding to implement, but produces results, which appear useful. Much further work will be required to develop this method for routine service, so that the individual assessments of case difficulty can be employed in the long-term monitoring of output quality. One simple method would be to normalize the quality score in a particular teleconsultation by dividing it by the difficulty level. However, it cannot automatically be assumed that a linear relationship is appropriate, and a more appropriate weighting scheme might require a logarithmic transformation of the difficulty level. Clearly, these matters all represent areas for future research.

The methodology proposed in the present work is perfectly general, and extends beyond telemedicine in high-resource settings to non-telemedicine work in conventional health care settings. Using a low-resource setting as the environment in which to develop a more general method represents a strength of the study, since it does not depend on a pre-existing, reliable, and efficient health care system to provide a foundation. Long-term quality assurance should assist those providing telemedicine services in low-resource settings to ensure that the services are operated effectively and efficiently, despite the constraints and complexities of the environment.

## **LIMITATIONS OF PROPOSED TECHNIQUE**

There are several limitations of the proposed technique (the direct estimation of case difficulty). First, the validity of the method must be established formally. Second, the optimum number of observers remains to be established. Both these matters stem from the sources of variability in the estimation problem being considered, where the underlying true value is obscured by variation between patients, by variation between observers, and also by variation between specialists (although the latter has not been examined previously in the present context).

Finally, the best method of combining the panel's scores requires some theoretical basis. Clearly, further research is required to investigate all this prospectively.

## **Table 3 | Direct assessment of case difficulty**.


a In each case, 0 = do not know was also an acceptable response.

## **CONCLUSION**

As telemedicine becomes adopted as a routine method of healthcare delivery, there is a requirement to implement quality assurance activities. However, there is little published information about quality assurance in store-and-forward networks, especially in low-resource settings. The present study builds on a previous proposal for measuring the quality of individual teleconsultations being produced by a network, and allows long-term process control by taking into account the difficulty posed by individual cases. The methodology is feasible and appears to produce useful results. It should assist those working in low-resource settings to ensure that telemedicine services are operated effectively and efficiently, despite the constraints and complexities of the environment.

## **ACKNOWLEDGMENTS**

We thank Will Wu for software support, and Professor Victor Patterson (Queen's University, Belfast) and Dr. Olivier Steichen (AP-HP, Tenon Hospital, Paris) for helpful comments.

## **REFERENCES**


google.co.uk/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&cad=rja&uact= 8&ved=0CCIQFjAA&url=http%3A%2F%2Fwww.health.qld.gov.au%2Fahwac %2Fdocs%2Fcet%2Fcomplexcarefull1.pdf&ei=0VrOU8GACceY1AXp34CACg &usg=AFQjCNFIPAouzwPFGjppYAAY-kHQkGsLlg&bvm=bv.71198958,d.d2k


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 08 August 2014; paper pending published: 30 September 2014; accepted: 12 October 2014; published online: 28 October 2014.*

*Citation: Wootton R, Liu J and Bonnardot L (2014) Assessing the quality of teleconsultations in a store-and-forward telemedicine network – long-term monitoring taking into account differences between cases. Front. Public Health 2:211. doi: 10.3389/fpubh.2014.00211*

*This article was submitted to Public Health Education and Promotion, a section of the journal Frontiers in Public Health.*

*Copyright © 2014 Wootton, Liu and Bonnardot. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited andthatthe original publication inthis journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

METHODS ARTICLE published: 24 November 2014 doi: 10.3389/fpubh.2014.00247

## Quality assurance of teleconsultations in a store-and-forward telemedicine network – obtaining patient follow-up data and user feedback

## **RichardWootton1,2\*, Joanne Liu<sup>3</sup> and Laurent Bonnardot 4,5**

<sup>1</sup> Norwegian Centre for Integrated Care and Telemedicine, University Hospital of North Norway, Tromsø, Norway

<sup>2</sup> Faculty of Health Sciences, University of Tromsø, Tromsø, Norway

<sup>3</sup> Médecins Sans Frontières (MSF) International, Geneva, Switzerland

<sup>4</sup> Department of Medical Ethics and Legal Medicine (EA 4569), Paris Descartes University, Paris, France

<sup>5</sup> Fondation Médecins Sans Frontières, Paris, France

#### **Edited by:**

Connie J. Evashwick, George Mason University, USA

#### **Reviewed by:**

Armin D. Weinberg, Life Beyond Cancer Foundation, USA Jay E. Maddock, University of Hawaii at Manoa, USA Frederick North, Mayo Clinic, USA

#### **\*Correspondence:**

Richard Wootton, Norwegian Centre for Integrated Care and Telemedicine, University Hospital of North Norway, PO Box 6060, 9038 Tromsø, Norway e-mail: r\_wootton@pobox.com

User surveys in telemedicine networks confirm that follow-up data are essential, both for the specialists who provide advice and for those running the system. We have examined the feasibility of a method for obtaining follow-up data automatically in a store-and-forward network. We distinguish between follow-up, which is information about the progress of a patient and is based on outcomes, and user feedback, which is more general information about the telemedicine system itself, including user satisfaction and the benefits resulting from the use of telemedicine. In the present study, we were able to obtain both kinds of information using a single questionnaire. During a 9-month pilot trial in the Médecins Sans Frontières telemedicine network, an email request for information was sent automatically by the telemedicine system to each referrer exactly 21 days after the initial submission of the case. A total of 201 requests for information were issued by the system and these elicited 41 responses from referrers (a response rate of 20%).The responses were largely positive. For example, 95% of referrers found the advice helpful, 90% said that it clarified their diagnosis, 94% said that it assisted with management of the patient, and 95% said that the telemedicine response was of educational benefit to them. Analysis of the characteristics of the referrers who did not respond, and their cases, did not suggest anything different about them in comparison with referrers who did respond. We were not able to identify obvious factors associated with a failure to respond. Obtaining data by automatic request is feasible. It provides useful information for specialists and for those running the network. Since obtaining follow-up data is essential to best practice, one proposal to improve the response rate is to simplify the automatic requests so that only patient follow-up information is asked for, and to restrict user feedback requests to the cases being assessed each month by the quality assurance panel.

**Keywords: telemedicine, telehealth, quality assurance, quality control, LMICs**

## **INTRODUCTION**

Follow-up is an integral part of consultation in medical practice. No doctor would give advice about a patient without attempting to follow the patient's subsequent progress and/or trying to obtain some feedback. This basic principle is not altered when the consultation takes place at a distance (teleconsultation). Follow-up is part of routine clinical care, conducted in order to confirm that the situation is evolving as expected, and to allow the diagnosis, prognosis, and treatment to be adjusted as appropriate. It is also important for doctors to learn from their successes and mistakes, as part of a reflective practice (1).

Thus, it is not surprising that surveys in telemedicine networks show that the specialists who provide advice wish to receive followup data about the cases they have worked on. In a survey of telemedicine users in Médecins Sans Frontières (MSF), almost all specialists wanted follow-up information (52% considered follow-up desirable and 47% considered it necessary or mandatory) (2). In a survey of specialists in the Swinfen telemedicine network, 83% stated that they would like to receive follow-up information about the patient (3). We assume that provision of follow-up data is useful in keeping the specialists motivated, i.e., to ensure their continued participation in the telemedicine network and their availability to provide advice. It is also probably the only way that specialists can improve their service, since many of them will be based in high-income countries and without feedback it is impossible for them to know if their answers are useful; prompt feedback from the referrer may be perceived as a mark of gratitude for the service provided, which is important since many specialists volunteer their time and expertise for free. While it can reasonably be assumed that the provision of follow-up data is useful for many reasons, there is no literature about this (an experiment to test the assumption would be difficult, although not impossible).

Follow-up is also useful for those running the network, especially if a research study is to be conducted. Follow-up provides information about the value of the telemedicine consultations, and about the performance of individual specialists. Information about the latter is very valuable for the case coordinator in the allocation process, since experience shows that some specialists answer more quickly and comprehensively than others. Finally, providing follow-up data is probably a good discipline for the referrers, as it makes them think about the progress of their patients and about the value of the telemedicine advice they have received.

In the present paper, we distinguish between patient *follow-up*, which is information about the progress of a patient and is based on outcomes,and user*feedback*,which is more general information about the telemedicine system itself, including user satisfaction and the benefits resulting from the use of telemedicine.

## **OBJECTIVES**

The primary research question was whether a method could be developed for obtaining follow-up data automatically in a general teleconsulting network, which was providing a service in lowresource settings. The secondary research question was whether it was feasible to obtain both follow-up data and user feedback simultaneously.

## **METHODS**

The present study required the development of a method to obtain data from the referrers and then a demonstration of its feasibility in practice. We combined the collection of both kinds of information into a single questionnaire, i.e., it represented a progress report.

The work was performed in two stages:


Ethics permission was not required, because patient consent to access the data had been obtained and the work was a retrospective chart review conducted by the organization's staff in accordance with its research policies.

## **DEVELOPMENT OF THE QUESTIONNAIRE**

A questionnaire was developed by a consensus between three experienced telemedicine practitioners (two were medical specialists with field experience). It was based on accepted tools used in previous studies (3, 5). The final questionnaire was evaluated and approved by an independent evaluator.

The final questionnaire consisted of 12 questions, which concerned both patient follow-up and user feedback, **Table 1**. The questions about follow-up concerned the referrer's opinion about whether the eventual outcome would be beneficial for the patient. The questions about feedback concerned the referrer's opinion about whether the process was satisfactory (e.g., the way that the referral had been handled in the telemedicine network) and what the benefits were, for the patient and doctor.

#### **Table 1 | Progress report questions**.


The questions concern follow-up (patient outcomes) or user feedback. User feedback encompasses satisfaction with the service (S) and benefit to the patient (Bp), the doctor (Bd), and the organization (Bo).

## **AUTOMATIC REQUEST FOR INFORMATION**

Modifications were made to the telemedicine system so that automatic requests for progress reports were sent to every referrer at a pre-determined interval after a new case had been submitted. The request allowed the referrer to log in to the server and then provided a link for the referrer to respond to the questionnaire.

#### **DEMONSTRATION OF FEASIBILITY**

To demonstrate the feasibility of the proposed approach, automatic requests for progress reports were issued in respect of cases submitted in the MSF telemedicine network for a 9-month period starting in October 2013. An email request was sent automatically by the telemedicine system to each referrer exactly 21 days after the initial submission of the case. When the referrer completed the progress report, an email notification was sent

#### **Table 2 | Summary of 41 responses**.


simultaneously to the expert(s) involved in the case and to the case-coordinators.

## **ANALYSIS OF RESPONSES**

Responses to the requests were analyzed approximately 4 weeks after the final request had been sent. The free-text comments were examined and, based on a content analysis, the main themes were extracted.

## **RESULTS**

## **ANALYSIS OF RESPONSES**

During the pilot trial, 201 requests for progress reports were issued by the system and these elicited 41 responses from referrers (a response rate of 20%). The responses were largely positive. For example, excluding the Do not know and Missing responses, 95% of referrers stated that they found the advice helpful, 90% said that it clarified their diagnosis, and 94% said that it assisted with management of the patient. In addition,95% said that the telemedicine response was of educational benefit to them. The responses are summarized in **Table 2**.

The qualitative analysis of the free comments confirmed this positive feedback from the responders, see **Table 3**. The expert advice was considered by the referrer as "clear, comprehensive, and useful," helping both in the clinical management (diagnosis and management) and the information delivered to the patient and relatives. Referrers considered that the non-availability of an investigation or treatment that had been suggested was the main limitation in following the advice received. For this reason, some referrers emphasized the importance of making the expert aware of the constraints of the referral setting and the limited resources available.

Satisfaction with the system was also very high and the words used by responders emphasized the efficiency of the system ("excellent, very good quality, quick, practical . . ."). In terms of benefit, avoiding unnecessary referral to a higher level of health care or avoiding further specialized consultation were mentioned as the main reasons for cost savings.

## **ANALYSIS OF NON-RESPONSES**

During the pilot trial, questionnaires were completed for 41 cases. That is, no questionnaire was completed for the other 160 cases. These two groups of cases might have differed in some way, and any difference might be a reason why the referrers decided to respond or not to respond. Various characteristics of the two groups were, therefore, compared. The median age of the patients in Group 1 (those with responses) was 27.5 years,

#### **Table 3 | Main themes in the free-text responses**.


(Continued)

## **Table 3 | Continued**


Note that one answer may include more than one theme.

## **Table 4 | Characteristics of the cases**.


## **Table 5 | Referrers who provided progress reports for all requests**.


Note that some cases were submitted by headquarters staff on behalf of field doctors in low-income countries.

and the median age of the patients in Group 2 (those without responses) was 22.0 years. However, the difference was not significant (*P* = 0.13). There were no significant differences in the gender of the patients in the two groups, nor the type of queries required to answer them, nor the number of queries for each case, see **Table 4**.

## **RESPONDERS AND NON-RESPONDERS**

Six referrers provided progress reports for every request they received, see **Table 5**. However, the majority provided either some

\*Age groups defined as: young 0–17 years; adult >17–60 years; older >60 years.

#### **Table 6 | Referrers who provided some or no progress reports**.


Note that some cases were submitted by headquarters staff on behalf of field doctors in low-income countries.

reports, or none, see **Table 6**. There were no obvious differences between the three groups (responders to all, some, or none of the requests) in the characteristics available for comparison, see **Table 7**.

## **DISCUSSION**

The present work shows that both patient follow-up data and user feedback information can be obtained in a telemedicine network, via an automatic questionnaire. In a 9-month pilot trial,



there was a response rate of 20%. How can we interpret this response rate? In physician surveys conducted in industrialized countries, a response rate of say 50–60% would be considered normal (6, 7). However, there is little published data about the response rate in online surveys of doctors in developing countries, and even less about the response rate in online surveys of doctors concerning the use of telemedicine in developing countries. A reasonable comparator is the study by Zolfo et al., of health-care workers using store-and-forward telemedicine in the management of difficult HIV/AIDS cases, which had a response rate of 19% (8).

The dangers of a low response rate are non-response bias (if the answers provided by respondents differ from the potential answers of those who do not answer), and response bias (if respondents tend to give answers that they believe that the questioner wants). Analysis of the characteristics of the referrers who did not respond, and the cases, did not suggest anything different about them in comparison with referrers who did respond. The comparison of referrers was, however, limited by the restricted information available about them. For reasons of information security, the telemedicine system stores little personal information about the users, and the accounts tend to be used by more than one person as staff are rotated through the field.

We were not able to identify obvious factors associated with a failure to respond. The response rate may, therefore, simply reflect the pressures of working in low-resource settings, and especially, the high turnover of field staff, which acts against the treating doctor being in post when a request for follow-up data is made some weeks later.

Measures to increase survey response rates are reasonably well understood, and include offering financial incentives, and following up online requests with copies of the survey sent out on paper. These are probably not appropriate in the present context. Nonetheless, it would seem prudent if this technique is to be adopted into routine service to try and increase the response rate. This raises a number of questions for future research:


As mentioned in the Introduction, it is highly desirable to obtain follow-up data for each case. Even though there are other ways to obtain follow-up information, e.g., from the regular dialog between expert and referrer, the benefit of using an automatic request is that a standardized report is obtained for each case. Thus, the main problem in practice is the low response rate, and how best to encourage the referrer to complete the questionnaire. One potential way to increase the response rate would be to reduce the number of questions, in order to allow the referrer to answer within 1–2 min. As shown in a previous survey (2), the main reasons given for not answering were a lack of time > forgotten to update > patient lost to follow-up > difficulties with Internet access (**Table 8**). This is why we propose to separate the reporting of follow-up data from obtaining user feedback.

If user feedback is solicited separately from the follow-up data, then a natural time to request it would be when the monthly quality assurance (QA) review is conducted (9). This activity involves an expert panel making an assessment of a recent case that has been selected at random. If user feedback is requested from the referrer for the same case, then both the panel's and the referrer's views on the quality of the teleconsultation can be compared.

Finally, it is worth noting that specialists tend to underestimate the value of their responses. In a recent survey (3), Patterson examined the perceived value of telemedicine advice. There were 62 cases where it was possible to match up the opinions of the referrer and the consultants about the value of a specific teleconsultation. In 34 cases (55%), the referrers and specialists agreed about the value. However, in 28 cases (45%), they did not; specialists markedly underestimated the value of a consultation compared to referrers. A survey of MSF telemedicine users found a similar phenomenon (2). This reinforces the importance of obtaining user feedback from the referrers, who are best placed to evaluate the benefits to the patient.

## **Table 8 | Data from a previous survey,\* (A) responses from referrers; (B) responses from specialists**.


\*Data from the MSF survey (50 questions) sent to 294 referrers and 254 specialists (in French and English) in December 2013 (2).

## **CONCLUSION**

Obtaining data from referrers by automatic request is feasible. The technique provides useful information for specialists and for those running the network. The modest response rate could be improved. Since obtaining follow-up information on each case is essential to best practice, a proposal to improve the response rate is to re-design the follow-up questionnaire to be as simple as possible, and to obtain user feedback separately, by sending a more detailed questionnaire in parallel with the randomly selected cases reviewed each month by the QA expert panel.

## **ACKNOWLEDGMENTS**

We thank Will Wu for software support and Dr Olivier Steichen (AP-HP, Tenon Hospital, Paris) for helpful comments.

## **REFERENCES**


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 06 August 2014; accepted: 07 November 2014; published online: 24 November 2014.*

*Citation: Wootton R, Liu J and Bonnardot L (2014) Quality assurance of teleconsultations in a store-and-forward telemedicine network – obtaining patient follow-up data and user feedback. Front. Public Health 2:247. doi: 10.3389/fpubh.2014.00247*

*This article was submitted to Public Health Education and Promotion, a section of the journal Frontiers in Public Health.*

*Copyright © 2014 Wootton, Liu and Bonnardot. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited andthatthe original publication inthis journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*