Challenges in MultiSite GxP Monitoring

Life science companies have a GxP requirement to prove that temperature and humidity-sensitive products have been protected from extreme environments to maintain product quality.  Usually this is accomplished with an automated monitoring system, such as the Vaisala viewLinc Monitoring System.

However, when a company has multiple sites, there are economies of scale that make it financially attractive to invest in an enterprise-level system.  An enterprise-level system is a single centralized server installation that houses one monitoring application that is accessed by multiple sites over the corporate LAN. A common example of enterprise systems is corporate email; the email server is in an IT center at a central corporate location, but employees can still send and receive email from different sites.

The centralization of a monitoring system and standardization on a single server-based software installation can provide some substantial benefits. The primary benefits are cost-savings and risk-mitigation.  On the cost savings side, only one server (or set of servers) is required, as well as fewer licenses.  Because only one server needs to be deployed and validated, implementation and administration costs are reduced.  In addition, having a single point-of-contact with a monitoring system vendor increases leverage for technical support and purchasing power. Risks are mitigated because the single server can be well protected in a corporate data center. Additionally maintenance activities such as back-ups, disaster recovery, and change controls are more easily implemented. 

There may sometimes be trade-offs with enterprise-level systems; for example, corporate solutions often require some compromise as different sites try to change their processes to work with the single solution. We can classify issues facing multi-site environmental monitoring systems into five broad types: Security, Administration, Geographic Variation, Multiple Site Types, and IT Infrastructure.

1.  Security: There is a general tendency towards privacy in GxP companies and that holds true for monitoring.  Nobody wants employees from other sites snooping into their area.  Practically, this is a real issue as a multi-site system does present opportunities for uncontrolled changes by folks from other sites.
    
2. Administration: With many sites operating on the same software installation, administration can get complex, much as it would with a single large installation, except there are more stakeholders.  As an example, some of our customers have over 4,000 sensors across their multi-site system.  If you know monitoring systems, you know this means at least 8,000 alarm thresholds to manage.
    
3. Geographic Variation: As sites get more widely dispersed, they begin to be located in different time zones than the corporate server installation.  Time-stamps are critical to monitoring systems, so collecting data from one time-zone, but storing it in another is a critical challenge.   We also see different local languages in use for our international customers.  Language matters as well, because it is always easier to work in your native language.
    
4. Multiple Site Types: Some multi-site systems involve different types of sites, ranging from warehouses and cold rooms, to laboratories, cleanrooms, and manufacturing suites.  Each site type, by definition, will have different requirements. This is easily seen when comparing the types of measurements needed at each site.  A cleanroom will have specific requirements for differential pressure measurements that just aren’t necessary for warehouses and cold rooms. In addition to these practical differences, a system that isn’t a good fit may be perceived as a “corporate solution,” and suffer from other problems simply because there is no sense of local ownership.
    
5. IT Infrastructure: An enterprise-level system allows simplifying the hardware at the server end of the system only.  The sensor end is more complex, as sensors need to be installed onto the LAN at each location.  This can present challenges if the local site infrastructure is not ready to accept multiple new devices.  There are often unexpected challenges such as physical installation of Ethernet drops, finding spare capacity on the network switches, bridging disconnected network segments, or insufficient bandwidth and signal strength if using wireless devices.

Before we offer up our take on solutions to these challenges, let’s look at how these challenges manifest themselves in different applications. There are three main types that face the challenges of multi-site monitoring: National Warehousers, Regional Hospitals, and Global Manufacturers.

These are an excellent fit for an enterprise-level system because they are usually following a business model that emphasizes centralization and duplication across sites. The warehouses can be spread over a region or nation and each site will often be very similar, measuring only ambient temperature and humidity, sometimes with cold storage or clean areas.  Most warehouses have a physically open architecture with few walls and barriers, which lends itself to easy deployment of IT infrastructure for sensor communication, whether wired or some form of wireless.  This customer will typically experience challenges 1 and 2 (Security and Administration) and occasionally 3 (Geographic Variation).

Example Vaisala Customers:

McKesson

Herbalife

Owens & Minor

YaoPharma

Alloga France

 The most common hospital application is monitoring of multiple small pharmacies within the hospital. When hospitals are involved in clinical studies or other GxP-regulated research, there are also laboratory areas that need monitoring.  Because hospitals can involve many departments and buildings, it is often more useful to view them as clusters of many small sites within one region or metropolitan area, or even within one building. 

Hospitals are unique in monitoring applications because there are many areas that are client-facing. These often exist in multiple different extant buildings that have often been connected to form a single continuous space. This provides unique issues with installation as the IT infrastructure is typically hard to access and overloaded by multiple use, especially when compared to an open warehouse.  This monitoring application type will typically experience challenges 1 Security), 2 (Administration), and often 5 (IT Infrastructure).

Example Customers:

Multi-site Research University

Lifesharing Tissue Bank

Associated Gastroenterology Medical Group (AGMG)

CTC North Clinical Research at University Hospital Hamburg
 

This application will experience all five of the challenges.  Monitored areas are spread over multiple countries and involve multiple local languages and time zones. They often include multiple site types performing very different manufacturing tasks.  The physical building architecture can be a challenge to monitoring system deployment. Remote areas can also find it difficult to access skilled tradespeople, making installation services a common challenge. 

Example Customers:

Teva Pharmaceuticals

AstraZeneca

Packaging Coordinators Inc.

Shanghai Fudan-Zhangjiang Bio-Pharmaceutical Co., Ltd.

Taken by themselves these challenges are not difficult to solve, but they are difficult to solve on an ad hoc or site-by-site basis.  With an enterprise-level system however, there is only one system and one system vendor to work with. On a practical level, this means that to provide a successful multi-site enterprise monitoring solution, the system's solutions to the common challenges must be built into the software by design and the vendor must support the customer in implementing the solutions.

First, the Security Problem.
How does a system provide privacy for each site so their operations cannot be viewed by the entire company?  More importantly, how can the system prevent users at one site from taking actions that affect local operations at other sites?  The solution is to have a segregation function that can hide one site from another.  Importantly, segregation must be sophisticated enough to involve security features, as well as to prevent users from making configuration changes without permission.

Access control must be built into the monitoring system's software architecture in the design phase. This was the solution designed into the Vaisala viewLinc software. In addition to the expected and usual security roles that define the actions a user is able to perform, viewLinc has another security layer that defines what items can be seen or impacted by their actions. This is done by a function called object-level security. In this way, viewLinc ensures that the user experience is virtually identical to that of a single-site system.

Second, the Administration Problem.
If you have ten sites on a multi-site system, you will have ten times more threshold limits and often,ten times more users to notify. This can cause problems for system administrators. The solution is to have some functions that simplify system management so that many tasks can be performed with few actions. However, the solution must be elegant enough that it doesn’t add complexity. Like the security solution presented earlier, simplicity in system administration cannot be added to a software upon implementation. It must be included in the software design. 

To solve this issue, the viewLinc system uses templates for the most commonly configured functions.  For instance, GMP refrigerators are commonly set up with alarm limits of 2°C and 8°C. Because this configuration will be shared by many refrigerators, we allow the users to make a 2°C/8°C Threshold Alarm Template that can then be assigned to all the refrigerators that share this configuration.  If there were ten refrigerators with this configuration, using the template would decrease your set-up time by a factor of ten. By standardizing common processes in the viewLinc software, we created more opportunities for template-driven configuration and management.  Templates are provided for users, work schedules, threshold alarms, alarm notification schema, notification formats, and comment functions.  Because the basic business processes behind monitoring are standardized, viewLinc takes advantage of configuration similarities in monitoring applications. The practical result of template-driven administration is that the amount of work necessary to configure and manage the system drops dramatically.

When template-driven management is combined with Vaisala’s object-level security, this allows delegation of the administrative duties for the local instance of the system.  Now the local administrator can share the workload, while implementing standardized template tools.

Third, Geographic Spread.
As more monitoring sites become distributed internationally, they begin to cover disparate time-zones and include multiple local languages.  Each site will need to view data from their monitored areas in their own time zone. Additionally, users generally prefer to interact with a system in a familiar language, especially for review and administrative tasks that involve more than raw values of time and temperature and/or humidity. This is perhaps the clearest and most understandable example of a requirement to localize languages in software. 

This isn’t a big problem for regional organizations, like hospitals, or even many national organizations like warehousers.  But it is a big problem for international manufacturers whose operations often spread across multiple time zones and languages.To solve this problem, most monitoring system providers rely on the customer to use a procedural solution. This entails the customer standardizing on a single time zone and language. This means data time-stamps will only reflect the time-zone of the corporate data center.  Further, some users will be forced to use the system's standard language. The obligate conversion to a standard language and time presents an obvious set of risks.  In the case of language, there may be a simple misunderstanding.  In the case of time, it is quite easy for a conversion errors to occur when trying to describe a local deviation using a non-intuitive time zone.

The viewLinc software can automatically localize the user interface language and time-stamps to the settings used within the operating system of the client PC used to view the data. In addition, viewLinc software is offered currently in eight languages and as many time zones as are recognized by Microsoft Operating Systems. This means that viewLinc matches the language and time-zone parameters at the local site, removing a significant operational gap and source of error. In addition, these settings can be overridden by a user to configure their session to match the necessary international perspective.

Like the other solutions mentioned these types of solution cannot be applied as an afterthought.  Monitoring data is only relevant if the time-stamp of the data is known. Therefore, changes to time management in a database usually involves significant reorganization around how a fundamental data type is handled. Support for multiple languages in viewLinc also required some deep changes. To avoid syntax or grammar conflicts, the system has been designed to operate independently of the user interface language. Translations occur only for the GUI, through an independent translation table.  This creates an adaptable interface where new languages can be added easily, or translation errors corrected, all without having to touch the code of the software modules that drive the GxP functions of the application. This means the system is highly adaptable to users’ localization requirements, without compromising system functionality.

When a system is in the user's language and time zone, errors are reduced, and sense of system ownership increases, thereby decreasing system failures due to poor maintenance and lack of system knowledge. The risk can be mitigated in other ways, but making the software do the work, rather than a user, mitigates risks with a software-based solution.

Fourth, Different Site Types:
Some multi-site monitoring systems have different site types attempting to use the same monitoring system. Often this means that the system must provide a wide variety of measurement parameters. This is particularly a concern with Global Manufacturers. Warehouses usually measure only temperature and humidity.  In contrast the global manufacturer will need to monitor Differential Pressure in the cleanroom sites, to monitor water conductivity at laboratory sites, and to monitor ultra-low temperatures for sample storage in liquid nitrogen vessels at a research site.  The obvious concern here is that a single monitoring system will not have the built-in capability to monitor all the required parameters.  Often the work-around solutions used to bring data into the system can introduce unwanted risk and complexity.

The typical solution is to get a generalized monitoring system that collects most of the required data types, and then develop work-around strategies to import the more specialized data types into the database.  The primary problem with this approach is that custom programming is usually required to achieve data imports.  This creates more potential points of failure, as well as a significant increase in qualification effort to validate the custom code.

Vaisala chose an alternative approach with two solutions to the problem.  First, Vaisala manufactures a range of proprietary sensors, currently among the widest available from any monitoring system provider.  Second, Vaisala offers universal input loggers that accept analog current or voltage inputs from almost any external sensor. Data is brought into the viewLinc system through a standardized scaling and calibration process that removes the need for custom programming, keeping the validation simple.  For redundancy the data from the outside sensor is stored locally in the logger memory to reduce the risk of data loss.  This approach allows an almost infinite variety of data inputs to viewLinc, with no increase in system complexity.  This also allows customers to leverage their existing sensing instrumentation into a new viewLinc monitoring system.
 

Fifth and finally: IT Infrastructure Issues
One hidden challenge of implementing a monitoring system is finding ways to connect a large number of sensors to the LAN to communicate with the centralized server.  In warehouse applications this is typically easy due to the open spaces. However, it's more challenging in hospitals due to the closed nature of these spaces. Global manufacturers present their own challenges with a wide variety of sites, and a potential lack of local tradesman with the requisite IT skillset.  

Regardless the type of monitoring application, few customers have under-used Ethernet or Wi-Fi bandwidth available for a new monitoring system. Bandwidth use has a tangible cost and presents a challenge for any monitoring installation. The best approach is to allow for as many communication modalities as possible, so the customer can find the best fit for their existing infrastructure. These modalities typically include: Ethernet, Wi-Fi, GSM, and new emerging long-range RF technologies.

Traditional wired Ethernet connection is widely available and the most reliable form of connection.  The inherent reliability comes from the fact that there are few ways for a physical connection to be disrupted.  The tradition Ethernet connection can be enhanced for greater efficiency.  For example, using PoE (Power over Ethernet) to power a sensor or device can eliminate power wires and avoid costly utility requirements.  Furthermore, the use of multi-port Ethernet adaptors can allow multiple sensors to share an Ethernet connection.  Ideally your monitoring system vendor will have multiple Ethernet strategies to get the most versatility out of your existing Ethernet.

Wi-Fi was the first wireless monitoring solution that provided monitoring systems with a degree of freedom from the restrictions of a wired infrastructure.  It allows a sensor network to be installed with little infrastructure other than wireless gateways.  However, Wi-Fi comes with some inherent flaws, for example: networks may be difficult to keep secure without knowledgeable IT administration. And existing Wi-Fi networks are often over-utilized, or suffer from poor signal strength due to obstacle-rich environments.

Concerns about range and bandwidth drove the development of sensors that use GSM communication to increase range. This came with a downside: critical data must be sent outside of your private network to get to the central server. This is often too expensive and too risky for the typical GMP customer.

There has been much recent development into RF (Radio Frequency) technologies that have allowed for secure, long-range wireless communication from sensor to system.  These technologies are usually too limited for traditional high bit-rate networking purposes, but can be a perfect fit for environmental monitoring applications because typical monitoring data requires only low bit rate communication.  As a result, there are new long-range RF technologies available that have very low infrastructure requirements.  These technologies must be evaluated for long-range capability, secure data transmission, and low power requirements to ensure long battery life. These stringent requirements cannot be met by most RF technologies, like Bluetooth and Zigbee, due to insufficient range or high power requirements. 

One standout example of a suitable technology is the LoRa®  protocol, used by Vaisala in our new line of VaiNet Wireless Technology.  This new long-range wireless technology can achieve linear communication distances much further than Wi-Fi, with lower power requirements and lower cost. A significantly increased linear range (five times the distance or more, depending on the complexity of the environment)  means a single VaiNet Access Point can cover as much area as 20 traditional Wi-Fi gateways. This allows a virtually wire-free monitoring system, requiring only one Ethernet connection for up to 32 sensors.  The result is a viewLinc monitoring system that is almost free from infrastructure constraints.

IT Infrastructure issues are often a real challenge for deploying monitoring systems.  However, it is quickly becoming a problem with many monitoring applications because of a growing variety of new communication methods.  Our recommendation is to select a monitoring system provider with a range of devices that can communicate in many modalities.  In the Vaisala viewLinc system, we provide loggers that can communicate over Ethernet, Wi-Fi, and our proprietary VaiNet long-range RF protocol.

The cost savings, efficiency gains, and simplified compliance are compelling enough that a shift to enterprise monitoring solutions is almost inevitable, especially with the current trend of mergers and acquisitions. The overarching challenge in environmental monitoring applications lies in finding ways to ensure that the corporate one-size-fits-all solution will actually fit the needs of the various sites monitored.

Over the past years, we have identified five major challenges in multi-site monitoring across the more common application types. In our experience the single most effective step in addressing these challenges is to select the system that solves the most common problems with the greatest simplicity.

We have witnessed many customers overcome the most common multi-site monitoring issues with viewLinc in large part because the system was designed for multi-site use. When the software is designed with multi-site monitoring in mind, it is sufficiently adaptable in terms of architecture and options.

"Everything should be made as simple as possible, but no simpler."

Albert Einstein

Customer Comments:

“[The] system is easily scalable without extra costs, increases our efficiency with its remote reading abilities and ease of use, and the measurements are very accurate.”
- Mats Andersson, Project Manager, AstraZeneca

“It was important to us that the system could be deployed internationally and Vaisala was the only company we found that could support us throughout our other regions...”
- Gary Swanson, Senior Vice President of Quality for Herbalife International

“After years of working with the system, generating the kind of reports that make auditors happy, we’ve found Vaisala’s viewLinc Monitoring System to be bullet-proof.”
- Timothy Phelps, Facilities Engineering Manager, McKesson Specialty Distribution

We also have a recorded 1-hour webinar on this topic: Multisite Monitoring