In-memory analytics: Strategies for real-time CRM
We are on the brink of true “in-memory analytics,” a technology that will allow operational data to be held in a single database that can handle all the day-to-day customer transactions and updates as well as analytical requests — in virtually real time. To drive the shift to this new technology, CIOs must make sure the business understands its advantages and devise a governance strategy to manage its rollout and monitor its use.
In-memory analytics Strategies for real-time CRM
Atlanta Ralph Alewine Partner +1-404-519-0184 ralph.alewine @strategyand.pwc.com Berlin Florian Gröne Principal +49-30-88705-844 florian.groene @strategyand.pwc.com
Chicago Eduardo Alvarez Partner +1-312-578-4774 eduardo.alvarez @strategyand.pwc.com Düsseldorf Dietmar Ahlemann Partner +49-211-3890-287 dietmar.ahlemann @strategyand.pwc.com
Frankfurt Olaf Acker Partner +49-69-97167-453 olaf.acker @strategyand.pwc.com Milan Enrico Strada Senior Executive Advisor +39-02-72-50-93-00 enrico.strada @strategyand.pwc.com
Sydney Peter Burns Partner +61-2-9321-1974 peter.burns @strategyand.pwc.com
About the authors
Olaf Acker is a partner in Strategy&’s Frankfurt and Dubai offices. He focuses on business technology strategy and transformation programs for global companies in the telecommunications, media, and high technology industries. Dr. Florian Gröne is a principal with Strategy& in Berlin and New York. He works with communications, media, and technology industry players on defining their go-to-market strategies and operating models and on transforming customer-facing processes and enabling technologies. Adrian Blockus was formerly an associate with Strategy&. Dr. Carsten Bange is the founder and managing director of the Business Application Research Center (BARC), the leading market analyst for business intelligence and data management in central Europe. He consults regularly on business intelligence and data management strategy, architecture, and technology selection.
This report was originally published by Booz & Company in 2010.
For years, the process of devising customer data queries and creating business intelligence reports has been a lengthy one. That’s because the information needed must be pulled from operational systems and then structured in separate analytical data warehouse systems that can accept the queries. Now, however, we are on the brink of true “inmemory analytics,” a technology that will allow operational data to be held in a single database that can handle all the day-to-day customer transactions and updates as well as analytical requests — in virtually real time. The advantages of in-memory analytics are many: Performance gains will allow business users to retrieve better queries and create more complex models, allowing them to experiment more fully with the data in creating sales and marketing campaigns, and to retrieve current customer information, even while on the road, through mobile applications. The resulting boost in customer insights will give those who move first to these systems a real competitive advantage. Companies whose operations depend on frequent data updates will be able to run more efficiently. And by merging operational and analytical systems, with their attendant hardware and software, companies can cut the total cost of ownership of their customer data efforts significantly. To drive the shift to this new technology, CIOs must make sure the business understands its advantages in terms of better customer intelligence and lower overall cost. To do so, they must make a strong business case for the transformation — always a challenge with business intelligence systems — including ease of use, better analytical reports, and better decision making. And they must devise a governance strategy to manage the technology’s rollout and monitor its use.
Analytics on the fly
Consider the plight of an insurance sales rep sitting down with a customer to discuss changes to his life insurance policy. The only information he has about the customer may be months old, so pricing a new policy given the customer’s changing circumstances will have to wait until the rep can add the new information to the system back in the office. And because he can’t analyze the new data immediately, there’s no opportunity to offer the customer specific details on new products he might be interested in. Now, however, a new technology called “in-memory analytics” lets that sales rep enter new data into his company’s database from a tablet computer sitting on his lap as the meeting is taking place. In real time, he can analyze the customer’s new situation and generate current pricing information, as well as information about other products that might suit the customer’s current needs. As a result, the rep has a much better chance of up-selling the customer on his current policy and cross-selling him on other products. Until recently, large customer relationship management (CRM) systems depended on two separate databases: The operational database maintained the day-to-day, high-volume transactional data, while the analytical database took the data needed to perform specific customer analyses and stored it separately. As a result, it was impossible to run real-time queries against the most up-to-date customer data. Thanks to major advances in the speed, cost, and sophistication of storage and memory technology and in the power of processors, however, the promise of real-time analytics — through which business users can access the full set of operational data when creating their reports — is finally becoming a reality (see Exhibit 1, next page). By giving business users access to truly live customer data, in-memory technology will transform how companies analyze and use that data. As such, it offers three significant benefits over traditional data warehouses:
“In-memory analytics” lets a sales rep enter new data into his company’s database from a tablet computer.
Exhibit 1 An integrated CRM architecture can speed up analytics requests
Analytical CRM Mining Operational CRM Campaigns Campaigns Segmentation Sales
Real time Real time
Transactional data, analytics requests
Batch scores, models
Source: Strategy& analysis
• Performance improvements: Because users can now interact with and query data in memory, response time and calculation performance is dramatically improved. This increase in performance allows endusers to run more complex queries and gives them better modeling capabilities, adding up to greater business value (see “Inside the Technology,” next page). • Customer value creation: In-memory analytics gives business users instant self-service access to the information they need, providing an entirely new level of customer insight that has the very real potential to maximize revenue growth through more powerful up-selling and cross-selling. • Lower costs: Total cost of ownership is significantly lower compared to traditional data warehouses, in part because all the data is now stored in one place. And while in-memory technology allows for the analysis of very large data sets, it is much simpler to set up and maintain. Rapid departmental deployments can free up IT resources previously devoted to responding to requests for reports.
Inside the technology
Until very recently, the effort to create, store, and analyze critical transactional data related to all kinds of business activities was a cumbersome and expensive process. Operational data — the high-volume, transaction-heavy data generated through a variety of business processes, including sales, order management, and customer care operations such as call centers — was maintained in huge data warehouses to ensure reliable performance and data integrity. And because the sources of the operational data typically varied significantly, maintaining it all in a single database with the homogeneous data model that could serve as a “single point of truth” proved very beneficial. Meanwhile, the analytical data used to gather customer and performance insights, to segment customers, and to model and predict future behavior through customer usage and payment history, for example, was typically drawn periodically from the operational database and maintained separately. As valuable as that analytical data proved in boosting customer profitability and allowing more efficient up-selling and cross-selling, the architecture had some very real downsides. Because it had to be duplicated periodically, the data in the analytical data warehouse was frequently at least a day or two — and sometimes as much as a month — out of date. A specially designed data mart had to be built for practically every new analysis request, which meant long deployment cycles, low project success rates, and evergrowing data volumes at ever-higher cost. And the process introduced an additional layer of complex analytical software into the enterprise architecture, requiring additional training. Typical business users could only generate predefined standard analytical reports; anything more complex needed to be set up by a handful of power users. Now, however, this long-time separation between operational and analytical databases is finally coming to an end. With the emergence of multicore processors, increasing clock speeds, and 64-bit technology, combined with the rapid decrease in the price of memory, data can be managed entirely in main memory. While the idea of managing data in memory is not new, efforts to do so were hampered until recently by the fact that the old 32-bit architecture could address only 4 gigabytes of memory, and processors were not fast enough to give in-memory databases any real performance advantage. But with new ways of organizing, buffering, and accessing the data, the performance improvements are significant (see Exhibit A, next page). The capacity capabilities of these systems are now equaling those of large disk-based databases. For example, a pilot implementation of a 40-terabyte in-memory database was recently completed, and theoretically, databases as large as 16 exabytes (16,384 petabytes) could be managed with inmemory technology, based on today’s architecture. Throughput is seven times higher, and the response time is virtually instantaneous.
Exhibit A In-memory technology offers vastly superior response time and throughput
Throughput (in thousands of transactions per second)
5x 700 600 500 400 300 200 100 0
Update Select Source: Strategy& analysis
Response time (in microseconds)
120 100 80 60 40 20 0
On-disk database In-memory database
Several factors — involving improved analytical speed and performance and better analytical results — are driving the push to in-memory technology at the enterprise level. Business demand The delays that typically arise out of the periodic extraction, transformation, and loading (ETL) of data from the operational to the analytical systems may be generally acceptable in doing trend analysis and forecasting. But traditional data warehouses simply cannot keep pace with today’s business requirements for fast and accurate analytical data, especially in situations where mobility is becoming the norm. In every industry, customers now expect instant responses to their requests and questions; in this environment, inmemory technology allows companies to create an entirely new level of customer experience, and it gives users instant access to the data they need to provide online self-service, real-time customer segmentation and dynamic pricing. Time-sensitive industries like airlines and transport logistics will now have access to real-time information in running their operations, and the resulting increase in efficiency will become a significant competitive advantage (see Exhibit 2, next page). Performance of analytics Most analytical applications have moved beyond the spreadsheets and tables offered by traditional reporting tools and now use interactive data visualization as the end-user interface, which allows many more people in the organization to make use of these systems. However, the new interfaces, which offer users interactive dashboards and the ability to perform much more intuitive tasks, demand very fast response times, as users now expect instantaneous results. Since in-memory analytics allows data to be accessed directly from memory, query results come
Traditional data warehouses simply cannot keep pace with today’s business requirements for fast and accurate analytical data.
Exhibit 2 Selected benefits of real-time analytics across different industries
Automated trading, online banking, huge volumes
Just-in-time people tracking
Exploding “look-to-book” ratios
– Optimization – Operations – CRM – BI
Value-add services, billing, subscriber database consolidation, fraud management
User authentication & authorization, credit check, best bets
Multichannel selling, cross-selling/ up-selling, personalization
Source: Strategy& analysis
back much more quickly than they would from a traditional disk-based data warehouse. The time it takes to update the database is also significantly reduced, and the system can handle many more queries (see Exhibit 3, next page). Growing data volumes The sheer amount of transaction data being digitally captured and stored is increasing exponentially, as are unstructured forms of data such as e-mail, video, and graphics. According to one estimate, 0.8 zettabyte of data was produced in 2009 — if a gigabyte were the size of a sesame seed, a zettabyte would equal the diameter of the sun — and that is expected to rise to 35 zettabytes by 2020. At the same time, tighter regulations involving the tracking of financial transactions and customer data put the onus on organizations to maintain this data and keep it available for years. Much of this data still resides on legacy systems, which are costly to operate and maintain. In-memory analytics allows such data to be accessed rapidly on an ad hoc basis, without having to build additional complex data marts and load data into them. Instead, these systems allow users to connect to legacy data stores, populate an ad hoc database, conduct the analysis, and then discard the in-memory database once the analysis is complete. Speed of deployment Given the rapid growth of data volumes and the proliferation of applications dependent on databases, companies are struggling to manage the many business intelligence (BI) efforts being developed throughout their organizations. In many cases, for instance, users simply want access to their specific transactional systems for reporting and analysis, without the need to deploy a full data mart. In-memory analytics removes the need to build complex performance layers such as multidimensional cubes within the data warehouse; instead, users can run their analytical applications directly against an in-memory performance layer.
Exhibit 3 Performance comparison of different database types
Seconds 100 1,000
Disk-based, memory-cached databases
Throughput (transactions per second)
Source: Strategy& analysis
In addition to the real gains in performance and speed offered by inmemory analytics, these new systems can significantly improve the quality of the business and customer intelligence they generate. And they can transform how that intelligence is delivered, and to whom. The benefits include the following: • Improved decision making: The ease of use of in-memory technology allows anyone in the organization, from business analysts to managers, to build their own queries and dashboards with very little technical expertise. Control over critical data shifts away from those who manage it to the stakeholders who own and use it, allowing them to make better business decisions. • Richer insight: The significantly greater processing speed and calculation performance of in-memory technology lets end-users develop richer, more complex models, enabling better customer segmentation and more powerful campaign planning in the CRM space, for example. The result is significantly greater business value for the system as a whole. • Increased efficiency: Converting to in-memory technology as a platform for analysis allows a whole technological layer to be removed from the enterprise architecture, reducing complexity and the infrastructure the traditional systems required. Furthermore, the source data has to be created or populated only once and then is immediately available for any kind of analysis. Consequently, organizations can operate at a higher level of performance, deliver more reports per hour, and free up capacity on the source systems for other operational purposes. • Self-service business intelligence: In-memory analytics allows any user to easily carve out subsets of the enterprise business intelligence environment for convenient departmental usage. Work groups can operate autonomously without affecting the central data warehouse workload. In addition, in-memory technology enables a much greater degree of ad hoc analysis within the organization and allows users to
In-memory technology enables a much greater degree of ad hoc analysis within the organization.
source data rapidly, build analytical applications, and conduct specific investigations. Once the analysis is no longer required, it can be disposed of easily. Quick response times and strong visual interfaces also enable mobile BI applications, which can be used by salespeople to gain a complete view of customers, based on realtime customer sales data, while on the road.
Steps for the CIO
The virtues of in-memory analytics are many, but as with any new technology, it is the responsibility of the CIO to make these virtues clear both to top management and to business users. Most important, in-memory analytics must be seen as part of a broader BI strategy that takes into account its overall business value and the underlying technology architecture, while remaining aware of the challenges inherent in every major new technology. The top priority is to educate the business as to the value and advantages of in-memory analytics, as well as to the costs and risks involved. In many organizations, business users of analytical tools have grown accustomed to relying entirely on the IT department to perform its “data magic in the basement,” a process that can take days. Moreover, users frequently avoid using traditional BI tools because of their inherent complexity and difficulty. With in-memory analytics, as we have seen, response time can be virtually instantaneous, and users have the ability to design their own queries. It is critical to make clear to the business that a significant portion of the value of in-memory technology lies in its ability to open up these bottlenecks and offer users greater access to fresh data and increased query flexibility. As part of the education process, CIOs should identify and point out particularly valuable business opportunities that in-memory technology offers. These might include the ability to go beyond traditional BI reports to create powerful applications such as what-if analyses, interactive filtering, and pattern discovery, all in an easy, intuitive fashion. These capabilities should be actively promoted in order to foster a highperformance decision-making culture. But to accomplish this, many of the organizational processes with which both business users and the IT department are familiar will need to be rethought — an effort that, like any major change, must be planned and executed carefully. Calculating the business case for any BI effort, traditional or otherwise, is difficult. The new technology will require a significant up-front investment in new storage hardware and software, and in the training needed for both the business and IT staffs to make the best use of it.
A significant portion of the value of in-memory technology lies in its ability to open up bottlenecks and offer users greater access to fresh data.
Moreover, companies will need to maintain their old warehouses as they implement and test the new system, an additional up-front cost that must be taken into account. It will also be necessary to conduct a thorough cleansing of all customer data to avoid contaminating the new system with bad information. Once such systems are installed, most companies will struggle to calculate the tangible cost-of-ownership benefits, such as overall infrastructure savings or lower administrative labor costs. In order to build a better business case, CIOs must gain an in-depth understanding of the different types of BI applications and user segments involved, as well as the extent of the administrative maintenance effort required. In-memory analytics can then be better integrated into the overall BI tool strategy and positioned to either replace or complement current BI solutions. The proper role of in-memory analytics in a company’s overall BI architecture is critical. Don’t try to convert the entire architecture to in-memory technology all at once. Instead, develop a thorough investment road map that includes both a plan for incorporating inmemory technology in the standard architecture when possible — in order to prevent business units from adopting it as part of a “rogue” IT effort — and a strategy for switching over to the in-memory technology on a department-by-department basis. Building a governance strategy that can effectively manage the potential explosion in the number of analytical applications is essential. Such a strategy should include an inventory of analytical applications that clearly defines owners and use cases and that can serve as the basis for a wider rollout of in-memory analytics throughout the organization. An established “BI competence center” with the authority to drive standardization and exercise governance will be invaluable.
• End-users consistently rank slow query performance as among the top three concerns that affect their perception of the value of business intelligence systems. • In-memory analytics offers a vast improvement in process speed, query quality, and customer insight over traditional operational/analytical customer data warehouse systems. • The shift to in-memory technology will be driven by demand from the business for real-time customer and operational information. • CIOs must develop a strong business case for implementing in-memory analytics, including the new business opportunities it will enable and its advantage in terms of total cost of ownership.
By combining operational and analytical databases into a single instantly available warehouse, in-memory analytics will give business users access to a whole new realm of crucial customer information, transform how they use that information, and thus give them a real competitive advantage in the race to gain better customer insights more quickly. As with any business intelligence effort, however, the new technology’s virtues must be sold to business users, and its use must be monitored and managed carefully to ensure that all users are getting the most out of it.
Strategy& is a global team of practical strategists committed to helping you seize essential advantage. We do that by working alongside you to solve your toughest problems and helping you capture your greatest opportunities.
These are complex and high-stakes undertakings — often game-changing transformations. We bring 100 years of strategy consulting experience and the unrivaled industry and functional capabilities of the PwC network to the task. Whether you’re
charting your corporate strategy, transforming a function or business unit, or building critical capabilities, we’ll help you create the value you’re looking for with speed, confidence, and impact.
We are a member of the PwC network of firms in 157 countries with more than 184,000 people committed to delivering quality in assurance, tax, and advisory services. Tell us what matters to you and find out more by visiting us at strategyand.pwc.com.
This report was originally published by Booz & Company in 2010.
© 2010 PwC. All rights reserved. PwC refers to the PwC network and/or one or more of its member firms, each of which is a separate legal entity. Please see www.pwc.com/structure for further details. Disclaimer: This content is for general information purposes only, and should not be used as a substitute for consultation with professional advisors.