IDG Contributor Network: How in-memory computing drives digital transformation with HTAP
In-memory computing (IMC) is becoming a fixture in the data center, and Gartner predicts that by 2020, IMC will be incorporated into most mainstream products. One of the benefits of IMC is that it will enable enterprises to start implementing hybrid transactional/analytical processing (HTAP) strategies, which have the potential to revolutionize data processing by providing real-time insights into big data sets while simultaneously driving down costs.
Here’s why IMC and HTAP are tech’s new power couple.
Extreme processing performance with IMC
IMC platforms maintain data in RAM to process and analyze data without continually reading and writing data from a disk-based database. Architected to distribute processing across a cluster of commodity servers, these platforms can easily be inserted between existing application and data layers with no rip-and-replace.
They can also be easily and cost effectively scaled by adding new servers to the cluster and can automatically take advantage of the added RAM and CPU processing power. The benefits of IMC platforms include performance gains of 1,000X or more, the ability to scale to petabytes of in-memory data, and high availability thanks to distributed computing.
In-memory computing isn’t new, but until recently, only companies with extremely high-performance, high-value applications could justify the cost of such solutions. However, the cost of RAM has dropped steadily, approximately 10 percent per year for decades. So today the value gained from in-memory computing and the increase in performance it provides can be cost-effectively realized by a growing number of companies in an increasing number of use cases.
Transactions and analytics on the same data set with HTAP
HTAP is a simple concept: the ability to process transactions (such as investment buy and sell orders) while also performing real-time analytics (such as calculating historical account balances and performance) on the operational data set.
For example, in a recent In-Memory Computing Summit North America keynote, Rafique Awan from Wellington Management described the importance of HTAP to the performance of the company’s new investment book of rRecord (IBOR). Wellington has more than $1 trillion in assets under management.
But HTAP isn’t easy. In the earliest days of computing, the same data set was used for both transaction processing and analytics. However, as data sets grew in size, queries started slowing down the system and could lock up the database.
To ensure fast transaction processing and flexible analytics for large data sets, companies deployed transactional databases, referred to as online transaction processing (OLTP) systems, solely for the purpose of recording and processing transactions. Separate online analytical processing (OLAP) databases were deployed, and data from an OLTP system was periodically (daily, weekly, etc.) extracted, transformed, and loaded (ETLed) into the OLAP system.
This bifurcated architecture has worked well for the last few decades. But the need for real-time transaction and analytics processing in the face of rapidly growing operational data sets has become crucial for digital transformation initiatives, such as those driving web-scale applications and internet of things (IoT) use cases. With separate OLTP and OLAP systems, however, by the time the data is replicated from the OLTP to the OLAP system, it is simply too late—real-time analytics are impossible.
Another disadvantage of the current strategy of separate OLTP and OLAP systems is that IT must maintain separate architectures, typically on separate technology stacks. This results in hardware and software costs for both systems, as well as the cost for human resources to build and maintain them.
The new power couple
With in-memory computing, the entire transactional data set is already in RAM and ready for analysis. More sophisticated in-memory computing platforms can co-locate compute with the data to run fast, distributed analytics across the data set without impacting transaction processing. This means replicating the operational data set to an OLAP system is no longer necessary.
According to Gartner, in-memory computing is ideal for HTAP because it supports real-time analytics and situational awareness on the live transaction data instead of relying on after-the-fact analyses on stale data. IMC also has the potential to significantly reduce the cost and complexity of the data layer architecture, allowing real-time, web-scale applications at a much lower cost than separate OLTP/OLAP approaches.
To be fair, not all data analytics can be performed using HTAP. Highly complex, long running queries must still be performed in OLAP systems. However, HTAP can provide businesses with a completely new ability to react immediately to a rapidly changing environment.
For example, for industrial IoT use cases, HTAP can enable the real-time capture of incoming sensor data and simultaneously make real-time decisions. This can result in more timely maintenance, higher asset utilization, and reduced costs, driving significant financial benefits. Financial services firms can process transactions in their IBORs and analyze their risk and capital requirements at any point in time to meet the real-time regulatory reporting requirements that impact their business.
Online retailers can transact purchases while simultaneously analyzing inventory levels and other factors, such as weather conditions or website traffic, to update pricing for a given item in real time. And health care providers can continually analyze the transactional data being collected from hundreds or thousands of in-hospital and home-based patients to provide immediate individual recommendations while also looking at trend data for possible disease outbreaks.
Finally, by eliminating the need for separate databases, an IMC-powered HTAP system can simplify life for development teams and eliminate duplicative costs by reducing the number of technologies in use and downsizing to just one infrastructure.
The fast data opportunity
The rapid growth of data and the drive to make real-time decisions based on the data generated as a result of digital transformation initiatives is driving companies to consider IMC-based HTAP solutions. Any business faced with the opportunities and challenges of fast data from initiatives such as web-scale applications and the internet of things, which require ever-greater levels of performance and scale, should definitely take the time to learn more about in-memory computing-driven hybrid transactional/analytical processing.
This article is published as part of the IDG Contributor Network. Want to Join?
Source: InfoWorld Big Data