SQL Server 2014 – New Features: xVelocity memory optimized columnstore index

Some customers or colleagues recently asked me about the SQL Server concept xVelocity and in particular the SQL Server 2014 feature new xVelocity memory optimized columnstore index. So I decided to write a few words on the dbi services blog.

What is xVelocity?

Introduced in SQL Server 2012, xVelocity is composed of two features:

•  xVelocity in-memory analytics engine
•  xVelocity memory-optimized columnstore index

xVelocity in-memory analytics engine

The xVelocity in-memory analytics engine is an update of VertiPaq – introduced in SQL Server 2008 R2.
VertiPaq is composed of PowerPivot for Excel 2010 and PowerPivot for SharePoint 2010.
The main goal is to achieve breakthrough performances in analytic queries.
To achieve this aim, the engine uses columnar storage, state-of-the-art compression, in-memory caching, and highly parallel data scanning and aggregation algorithms.

Since SQL server 2012, this feature supports Business Intelligence (BI) – self-service (PowerPivot) and corporate (tabular mode in SSAS).
In the first case with BI self-service, an employee in a company can directly use PowerPivot for Excel to integrate data from different data sources.
In the second case with BI corporate, BI developers or/and IT Pros can use SSDT (SQL Server Data Tools) to create Analysis Services Tabular projects and deploy them on a server so that users can interact with the consolidated Data via Excel and Power View.

XVelocity memory-optimized columnstore index

xVelocity memory-optimized columnstore index is a feature which enables high performance query processing especially in Data Warehouses and Data Marts.

Unlike “Standard” indexes, columnstore indexes store data in memory (as the name memory-optimized columnstore index suggests) and the storage organization is not row-oriented.

The characteristic is that data is stored column-wise in compressed form. In SQL Server 2012, the first step was very limited and the main problema is that we cannot UPDATE or INSERT in a table with Columnstore Index.

For example, to update a table with columnstore index, the way is to drop the columnstore index, perform the update operation and then rebuild the index … which is not fun at all!

Now, in SQL Server 2014, this is possible without dropping the index.:-D

How is it possible?

A new concept / term is introduce in SQL Server 2014 for columnstore: The deltastore.

Deltafile is introduced with the In-Memory OLTP feature and deltastore with the columnstore index.
Yes, SQL Server 2014 is a real delta force!:-D

The best way to define it is to give you the MSDN definition:
“A deltastore is a row-store table that holds rows until the number of rows is large enough to be moved into the columnstore.”
They are other similitudes with the in-memory OLTP feature – let’s explain this.

Insert process

SQL Server inserts rows into the deltastore.
Rows accumulated in the deltastore are stored in rowgroups. Each rowgroup holds from 102,400 to 1,048,576 rows.
When the maximum number of rows in the rowgroup is reached, this rowgroup is marked as “CLOSED”.
The “tuple-mover”, a background process finds the closed rowgroup and moves it into the columnstore.

A SELECT query is an addition between non-closed rowgroups in the deltastore and the columnstore.

Delete process

To delete a row, there are 2 cases:

• In the columnstore, the row is marked logically as deleted, but not physically deleted. A rebuild is necessary to definitively delete the row.
• In the deltastore, the row is really deleted logically and physically.

Update process

To update a row, we have 2 cases – just like for the delete process:

• In the columnstore, the process is a delete process and an insert process like for the In-Memory OLTP process…
• In the deltastore, the row is just updated

Here is the MSDN reference for more information.

Type of Columnstore Index

The SQL Server 2014 xVelocity memory optimized columnstore index is composed of two index types:

• Clustered columnstore index
• Non-Clustered columnstore index

Clustered Columnstore index

Clustered Columnstore index is used for query performance and data compression.

Be careful, a table with a clustered columnstore index cannot have another index. You have two ways to create an index:

• using a GUI

Select the database, go to the index menu, right-click and select create index and create Clustered Columnstore index:

You can see if you can create the index or not:

Before the creation of the index, you must delete the others:

Now, you can create the index:

• using script

Non-Clustered Columnstore index

This is the “old” columnstore index introduced in SQL server 2012.

Like for the clustered columnstore index, you have two ways to create the index:

• using GUI

Select the database, go to index menu, right-click and select Create index and create Non-Clustered Columnstore index.
You can see if you can create the index or not:

Prior to the creation of the index, you must select columns.

And you must delete your previous Clustered Columnstore index:

Now, you can create the index:

• using script

What are benefits of this new xVelocity memory optimized columnstore index?

Making the columnstore index more functional, we can say that the main benefit is this new clustered columnstore index with the ability to continue queries during the updating process without the need to drop and recreate the index.

In my opinion, xVelocity is a very interesting technologies for my customers!