ORCHA provides an objective and independent assessment of health and medical Apps. This is an advisory not regulatory service, but we do advise where regulatory issues may be important and should be considered further. It is however ultimately the responsibility of the developers and/or publishers of an App to ascertain its compliance with all relevant regulatory standards.
The ORCHA Baseline Review (“OBR”) is ORCHA’s first level of assessment and involves a detailed ‘desktop’ analysis of Digital Health solutions looking across all of the key areas of regulation and compliance. The OBR is largely undertaken proactively as part of ORCHA’s ongoing assessment and monitoring of the whole Digital Health market place and we review the most downloaded and most recently updated Apps and related Digital Health solutions across over 250 health and care categories and conditions.
The OBR is primarily an assessment of an Apps compliance with current standards, regulation and good practice (together “Standards”).
A standard is an agreed way of doing something. It could be about making a product, managing a process, delivering a service or supplying materials – standards can cover a huge range of activities undertaken by organizations and used by their customers.
”Standards are the distilled wisdom of people with expertise in their subject matter and who know the needs of the organizations they represent – people such as manufacturers, sellers, buyers, customers, trade associations, users or regulators.” (British Standards Institute)
They can be of regulatory significance or form non regulatory requirements or required best practice in a given jurisdiction or area.
The Standards we currently look at in the OBR are:
The OBR seeks to assess an Apps performance through its compliance with these Standards. Our Review is regularly updated to reflect changes in these Standards. The higher the ORCHA Score achieved the more compliant the App is with these Standards and vice versa.
Whilst a high scoring App is not guaranteed to be effective or safe, or a poorly scoring App is not necessarily ineffective or unsafe, it does mean that the relevant Developer has taken more or less care over the Apps compliance with these key Standards than other similar Apps. In the critical area of health and care, we believe that developers should take compliance with Standards extremely seriously.
Some Apps are technically medical devices and for class iia, iib and iii devices, require full assessment and Approval in the EU through processes overseen by the national regulatory bodies such as the MHRA in the UK and HPRA in Ireland and in other jurisdictions by similar regulatory bodies. An App of this nature should not be made available to the general public until it has been assessed and appropriately certified and any Apps that we identify as being a none certified medical device is excluded from our general search.
If you become aware of any inaccuracy in the information presented in our Reviews or have any other concerns, please report this to us immediately at firstname.lastname@example.org.
ORCHA are not promoting or recommending any particular Apps through this process but are providing impartial information about an Apps compliance with Standards and a mechanism for end users to easily identify those Apps that best meet those Standards and to check which ones don’t.
The OBR process starts with a weekly analysis of all the Apps available on the App Store or Google Play in the ‘health, wellbeing/fitness and medical’ sections. We then filter out Apps that are not available to use in a supported jurisdiction (currently the UK, the Republic of Ireland, the Netherlands and Estonia) and Apps that have not been updated for 18 months or more. Of the remaining Apps we organise these into over 250 health and care condition categories and queue them within their category by most downloaded
There are a huge array of different types of health and care Apps available with an equally wide array of functionality and sophistication. At ORCHA we believe that just because an App offers more functionality it doesn’t mean it is automatically better than a functionally simple App. Apps that for example simply provide information and guidance – similar to health ‘leaflets’ – can be very useful in certain circumstances and this isn’t impacted by their relative functional simplicity.
In order to assess and score these Apps in a way that enables a fair comparison of like for like solutions, ORCHA has devised a sequence of Levels to attribute to an App. These Levels (currently ranging between Level 0-4), are indicative of the area of focus an App has (wellbeing, general health or specific conditions) and the level of functional complexity and associated risk. The Levels are an important part of the ORCHA Baseline scoring system which adjusts between each level to reflect a shifting prioritisation from the user experience measures towards the data security and clinical assurance review domains (outlined below).
The current App Levels are detailed below
These apps are designed to support general wellbeing and have limited feature sets and don’t collect any data. Because they are not health focussed and do not collect data we only assess their compliance in the User Experience domain.
These apps are focussed on general wellbeing but they can collect data and they are likely to have a number of more advanced features. Because they are not health focussed we only assess their compliance in the Data and Security and User Experience domains
These apps are focussed on general health. They may collect data and they may have a number of more advanced features. Where they collect data we assess their data use policies and compliance with relevant standards. Because they are health focussed we also assess their compliance in the Clinical Effectiveness as well as the User Experience domains.
These apps can be focussed on general health or supporting specific health conditions. They may collect data and they may have a number of more advanced features. Where they collect data we assess their data use policies and compliance with relevant standards. Because they are health focussed we also assess their compliance in the Clinical Effectiveness as well as the User Experience domains.
These Apps can be focussed on general health or specific conditions and contain advanced and complex features that are subject to formal regulation. Where they collect data we assess their data use policies and compliance with relevant standards. Because they are health focussed we also assess their compliance in the Clinical Effectiveness as well as the User Experience domains.
As part of our evaluation of an Apps Level, we also capture all of the Apps key functions and features. We have currently over 14 features that we regularly check for and we are constantly updating this as Apps develop new features and functions. The Functional assessment also dynamically changes the lines of enquiry that our Reviewers follow during the Review proper again to ensure that the assessment is as tailored to the type of App as possible. For example an App that collects sensitive and identifiable data will trigger a range of more detailed questions in regards to its Data compliance than an App that doesn’t collect this type of data.
The functional capabilities all become filterable search elements in the ORCHA App Finder, allowing all users to tailor their searches appropriately to find Apps that actually do the things that they want support around. For example, you can search for an App that simply provides Information and Guidance or an App that provides relevant forums or networks of support.
The functions and Levels that align to them are listed below:
The Review focuses on the three Review Domains outlined below. Each of these Review Areas has been designed by relevant experts and consists of a series of objective (Yes/No) questions which should be capable of being answered by our Reviewers from information in the App, on the relevant App Store or on a supporting website.
Our reviewers are recruited and trained by ORCHA from a wide array of backgrounds and roles. They are not experts in any particular field of App assessment but rapidly become expert in interrogating Apps to answer the questions posed by our Review Development Team who are all experts in the relevant areas such as clinical, technical, regulation, user experience and design.
Our Expert Review Development Team are responsible for the questions our Reviewers are looking to answer. They set out very clearly what evidence the Reviewers need to find to answer a question affirmatively and they also determine the consequences of the answer to each question in terms of positive or negative points being awarded, which drives our scoring process.
Our Reviewers are guided through each review through our online Review Engine and this ensures that all the relevant questions are investigated. Where our Reviewers encounter scenarios that the Review Development Team haven’t specified, the Reviewers refer these back to the relevant member of the Review Development Team for guidance.
This combination of highly trained dedicated reviewers and our wider Expert Group in the Review Development Team means that we can review lots of Apps in a timely and cost effective manner.
If you’re interested in becoming an ORCHA Reviewer please contact us here
Our expert team can be found here and their expertise and experience covers the wide array of elements that we review against. Whilst we have a number of clinicians in the Expert Group and a wider network of healthcare professionals that support many elements of our work, the Baseline review does not involve any direct clinical evaluation.
Our review process looks for proof points across a wide spectrum of App characteristics. From a clinical evaluation perspective, it would be very difficult and costly to undertake detailed analysis of the clinical accuracy and effectiveness of all Apps given the vast array of clinical and health related conditions and issues Apps now support. Even for suitably expert clinicians, the evaluation of many elements of an App would require more than a simple ‘read through’ or user test. This type of evaluation is therefore not practical for a ‘baseline’ review of the hundreds of thousands of Apps that are out there.
We therefore use indicators of clinical quality as a proxy for this level of detailed analysis to gauge whether an App appears to have the key ingredients in the area of clinical safety and effectiveness. These include an investigation of whether there is a suitable qualified individual or body behind the App, consideration of whether an App should be and is treated as a Medical Device with appropriate CE marking, an assessment of the evidence base that the developer has presented to support any efficacy claims.
These easily and practically assessable elements provide in most cases a good indication of whether an App can be considered clinically safe and sound. It is not 100% accurate and there are many instances where an App has scored poorly in this section simply because the Developer has not included this detail. We have not yet come across the reverse scenario where a Developer has scored well in this section and subsequently been found to have significant clinical safety issues or concerns but as with all but the most stringent and resource intensive assessment processes, this could theoretically happen.
If our Reviewers cannot find evidence to support an answer from these sources they will assume that the App is not compliant with or capable of satisfying the relevant requirement.
There are often cases where an App is in fact capable of satisfying particular requirements but that this is not made clear in any obvious publicly available place. In these circumstances Users are left with uncertainty as to the real position and our view is that in this circumstance they should proceed with caution.
Because of the factual nature of this approach most of these questions establish definitively whether or not the App does or does not do something that is relevant to the specific domain in question. In a small number of areas however, the answer to the question relies on the information provided on the face of the App by the relevant developer or publisher, being accurate. This is because of the significant logistical challenges that would arise if each of these elements were to require specific validation.
This is a pragmatic solution which has been adopted by most serious players in this space. For example both Apple and Google use this method in their acceptance approaches for integration with Apple Health and Google Fit respectively. It is also the approach that is being pursued nationally by the team from NHS Digital, PHE, Nice and NHSE in their development of an NHS Digital App assessment approach.
The OBR dynamically adjusts as the type or characteristics of an App evolves (i.e. does it collect Data or not, is it Clinical, does it interoperate with third party systems etc). This dynamic adjustment is a crucial part of the process and recognises that there are many different types of health Apps and applying a blanket assessment model is highly inefficient and ineffective. The dynamic nature of the review also applies to the scoring system that also adjusts to the characteristics of an App and provides a weighted impact dependent for example on the sensitivity of the data that an App handles or the level of clinical guidance it offers.
What do we look for? - This Review Domain looks at what information an App collects from you, what it does with that information and how secure is its handling of that data. For some Apps this is only a minor issue as they don’t gather much information. Other Apps can gather lots of very personal and sensitive information and for these it is important to know how they go about handling this.
The Baseline Review looks at all aspects of data collection, handling and use and covers all the aspects of GDPR that are capable of being assessed on a desktop analysis. The scoring mechanism in this area is responsive to the type of data collected and any failures to comply with regulation or best practice are more heavily penalised depending on the nature of data held or managed.
In terms of data security, we currently seek evidence that all data gathered is securely held and transported. This aspect of our review does rely on App developer/publisher statements and in theory there is a risk that an App developer/publisher could be misleading their users and misrepresenting the real position. However, in our view - and the view of the likes of Apple, Google and the NHS Programme team - this is a minimal risk. There will of course always be unscrupulous developers, however they expose themselves to a wide array of legal liabilities within most of the major jurisdictions in which they operate, if they make statements that are clear misrepresentations. In our experience, most of the most misleading Apps have actually used the 'small print' to clearly identify that the claims they make about the apps are 'not to be relied upon' rather than the other way around.
This review area looks at what evidence there is that there is a suitably qualified professional or organisation behind the App or that it has been endorsed, accredited or underwritten by such an individual or organisation. What constitutes a suitably qualified professional depends upon the nature of the App. Clearly for a clinically focussed App we would expect to see a relevant clinician involved. This gives a sense of whether or not the App has appropriate professional input and guidance and aligns to one of the required standards of evidence for lower Tier Apps under the recently published Standards of Evidence publication by NICE.
We also assess what evidence there is around user testing and the effectiveness of the App in terms of claimed health benefits. Where evidence is available this is assessed by our SME team who are NIHR accredited assessors, who will look at the findings and methodology of any study or trial to ascertain whether this is robust evidence to support the claimed benefits.
Finally we assess whether an App should be viewed as a medical device under the current and emerging regulations and check to see if it is suitable certified and CE marked with an appropriate classification. If we find an App is likely to be viewed as a Medical Device and it isn’t suitably certified, we will exclude it from any direct search results. This is again an area where some aspects of this assessment are handed off to our SME team for more detailed assessment.
This Review Area looks at how an App has been designed to support the overall user experience. This includes its availability on both primary platforms, how customisable the App is, how well it caters for end users with different capabilities and needs – such as visual or other impairments – how content is managed, how well supported the App is and what it costs.
This area also looks at overall User Review scores from both platforms but we only apply a very small amount of the overall scoring mechanism to this aspect because of the difficulties in relying on this data.
All this analysis results in an overall ORCHA score which is built up from the answers to each of the questions in the three Review Domains. Some questions earn positive points and some earn negative points. The ORCHA Score aims to deliver a meritocratic evaluation with all Apps being treated equally and fairly irrespective of their current popularity or the financial position of their Developers.
Any score below 65% would indicate that an App has some issues that users should investigate further prior to using this App. Scores below 45% indicate that an App has considerable issues or challenges and in its current form is potentially unhelpful or unsafe.
The final stage of the process is Developer notification and publication stage. When the review is complete, we send a notification to the relevant developer/publisher giving them an opportunity to preview the Review ahead of its publication. This gives them an opportunity to highlight any obvious errors or issues before we release the Review onto all our platforms. Once the notification period has expired, we publish the Review on all our platforms.
All our reviews will remain valid until a new version of the App is produced. If a new version is not produced within 18 months of the current version, we mark the relating to the App as 'Out of Date' and the ORCHA App Score will start to degrade at the rate of 5% per month.
If the Developer implements a new version of this App we will automatically detect this via the relevant App Stores and we will mark the current review as being related to an old version of the App on all our platforms. The new version of your App will go back into the queue for a Re-Review and we will in time undertake this.