Blurring Lines: OLTP vs OLAP – Part 2

In recent years, several technologies have emerged that are slowly reducing the gaps and paving way for a consolidated system. Advancements in hardware and software areas has triggered this disruptive phenomenon.

In-Memory Databases

RAMCost of Memory, specifically RAM has gone down considerably in the last few years. Coupled with the price drop, new and faster storage medium based on Solid State Disks (SSD) was introduced. Physical hard drives were slowest and hence kept away from the core data and used as backup medium. All the above factors resulted in three layer memory architecture.

Hardware manufacturers have re-designed the traditional memory storage design to accommodate large volumes of physical memory (e.g. 512 GB, 3TB etc) as primary memory. Apart from highly expanded physical RAM, the secondary storage using SSD was tightly integrated. Storage Area Networks (SAN) or Hard disks acted as tertiary data location. Software scans the primary physical memory periodically to spot hot versus cold data and move them to different storage areas accordingly.

Problems regarding persistence of data are addressed by the SSD technology and robust backup functions. This new architecture with three different layers of memory ensured that most frequently used data was always available and resided in main memory.

Row & Column Store Merge

Row_ColumnThese are the Two prominent methods of storing data at the file system level in a database system. Row Store was heavily adopted in OLTP applications and stored the entire record contiguously. Column store was the best approach for OLAP applications, where data was grouped into chunks before writing into database. Each type of store data differs drastically and have their own advantages and dis-advantages. Both approaches are present for quite some time and were implemented by all leading database vendors. From a technical perspective, it requires drastically different logic to implement SCRUD operations between two store mechanisms.

Today, the data architect has high degree of control over Row & Column store methods. One can pick and choose different store methods for individual database tables based on analysis and performance. Switching between the two storage mechanisms on the fly as well as maintaining copy of a single table in both store formats are supported now. This technology has enabled a single system to perform dual role based on the type of data and task it is handling.

NOTE: SAP HANA is a good example of this disruption and a similar system from house of Oracle is the Exalytics In-Memory machine. IBM and HP have their own product lines too in this arena.

1 Trackback / Pingback

  1. Star vs Snowflake | "Data" to "Wisdom" Journey…..

Comments are closed.