Big Data - dbi Blog

Exploring the Future of Data at SQLBits 2025 in London

Hocine Mechara — Sat, 21 Jun 2025 12:02:05 +0000

I’m looking back on three amazing days of interesting technical insights from the SQLBits event 2025 in London. It’s actually my first time having the pleasure of attending one of Europe’s biggest data platform events and I can sincerely say, that I enjoyed every minute of interesting workshops and sessions all around Data management, Data engineering and analytics with the latest technologies from Microsoft.

The sessions and workshops on the event mainly focused on the evolution of AI and a lot about Business Intelligence and Data Engineering with a big focus on Microsoft Fabric the unifying data platform from Microsoft. The choice of topics reflects the direction in which I believe the industry is heading, and I took the opportunity to gain valuable insights from renowned experts.

On the first day, I attended a one-day workshop on the latest AI functionalities in SQL Server and Azure SQL, focusing on how these features support the integration of external AI models with your data, enabling the development of context- and data-aware AI assistants for your application’s users based on the RAG (Retrieval-Augmented Generation) pattern. It was very interesting, the workshop was delivered in a high quality and it was even my personal highlight of the event.

On the second day I joined again a one-day workshop about Microsoft Fabric. I was very curious about Microsoft Fabric because I heard a lot of advantages about this end to end SaaS Data Platform and I was happy to learn more about it. The workshop covered a lot of topics regarding Microsoft Fabric and at the end of the day I had a good feeling of understanding the concept and the core elements about Microsoft Fabric.

During the break I enjoyed a fancy coffee served at the Microsoft booth

On the last day I attended again a one-day workshop about Microsoft Fabric but with more focus on the administration of a Fabric Tenant including workspace management, security, Git Integration etc. The workshop was again delivered in a good quality and I really enjoyed to attend. The Kahoot Quiz at the end of the workshop confirmed my Impression that I’m on track of understanding this platform quite well.

The event was awesome, I learned a lot about the latest technologies and I hope that I will be able to apply the gained knowledge in upcoming projects. I can highly recommend this event to every data professional working with Microsoft technologies!

L’article Exploring the Future of Data at SQLBits 2025 in London est apparu en premier sur dbi Blog.

Oracle Technology Roundtable for Digital Natives – Let’s have a look at AI, Cloud and HeatWave

Oracle Team — Fri, 07 Mar 2025 08:16:19 +0000

Yesterday I participated to the Oracle Technology Roundtable for Digital Natives in Zurich.

It was a good opportunity to learn more about AI, Cloud and HeatWave with the focus on very trendy features of this product: generative AI, machine learning, vector processing, analytics and transaction processing across data in Data Lake and MySQL databases.

It was also great to share moments with the Oracle and MySQL teams and meet customers which gave feedback and tips about their solutions already in place in this area.

I’ll try to summarize below some key take-away of each session.

Unlocking Innovation: How Oracle AI is Shaping the Future of Business (by Jürgen Wirtgen)

AI is not a new topic. But how do we use it today and where are we in the process, early or advanced?

To answer this question, you can have a look to the stages of adoption:

Consume (AI embedded in your applications) -> SaaS applications
Extend (models via Data Retrieval, RAG) -> AI services
Fine tune -> Data
Build models from scratch -> Infrastructure

AI is not AI. The best AI starts with the best data, securely managed. Which can be translated back into a simple equation: Best Data + Best Technology = Best AI.

Innovations in HeatWave & MySQL – The Present and the Future (by Cagri Balkesen)

HeatWave is an in-memory query processing accelerator for data in MySQL transactional RDBMS or data in Object Store in different format.

Normally you need to put in place and maintain ETL processes to produce data that can be used effectively by analytics. This brings several drawbacks:

Complexity
You have to maintain different systems, on which you’ll have security issues to handle and costs to assume.

Using HeatWave, you don’t need that anymore, because it’s a single platform which allows you to manage together all your OLTP, OLAP, Machine Learning and GenAI workloads.

Which are the advantages of using HeatWave?

You current SQL syntax doesn’t need to change
Changes to data are automatically propagated to HeatWave
Best performances for your queries
Efficient processing for Data Lake
Best platform for MySQL workloads
Built-in GenAI & Vector Store
Available in multi cloud (natively on OCI, it can run inside AWS, you can setup a private interconnection for Microsoft Azure, and there are works in progress for Google Cloud).

HeatWave is based on a massively parallel architecture which uses partitioning of data: each CPU core within a node processes the partitioned data in parallel.

Driven by Machine Learning algorithms, HeatWave Autopilot offers several features such as:

Improvements in terms of performance and scalability
Provisioning, data loading, query execution and fault management automation, to reduce human errors.

Finally, according to Oracle, with HeatWave you will have best performances and lowest costs rather than competitors: Snowflake, Amazon Redshift, Google BigQuery and Databricks.

Building Next-Gen Applications with Generative AI & Vector Store (by Adi Hochmann)

As we said, Oracle HeatWave allows you to manage together all your OLTP, OLAP, Machine Learning and GenAI workloads.

Steps to build a GenAI application are the following ones:

Create a vector store
Use vector store with LLMs

And this is performed using the following routines:

call sys.HEATWAVE_LOAD (…);

call sys.ML_RAG(@query,@output,@options);

But how to train data for Machine Learning? The process and tasks done by a data analyst could be complex and this is replaced here by the AutoML:

CALL sys.ML_TRAIN(‘data_source’, ‘model_type’, JSON_OBJECT(‘task’, ‘classification’), @result_model);

This is useful for some use-cases such as classification, anomalies detection, recommendations, predictive maintenance, …

Additional tip: Adi used MySQL Shell for VS Code to run his demo. This extension enables interactive editing and execution of SQL for MySQL databases and MySQL Database Service. It integrates the MySQL shell directly into VS Code development workflows and it’s pretty nice!

Oracle Cloud for Digital Natives: Supporting Innovation and Growth (by Claire Binder)

Which are the 5 reasons why Digital Natives picked OCI?

Developer-First openness and flexibility, to speed acquisition
Advanced Data & AI Services, to achieve innovation and agility
Technical and global reach, to achieve scalability
Security, compliance and resilience, to control risks
Cost efficiency and TCO, to achieve optimized spending.

Linked to the point 4, there are several services in OCI to avoid data breaches in terms of prevention, monitoring, mitigation, protection, encryption and access.

To select a Cloud provider, the recommendation would be to choose a solution which allows you to run converged open SQL databases, instead of single-use proprietary databases.

And finally, Oracle brings AI to your data with his new 23ai release and some of its features, such as Property Graphs and AI Vector Search.

Analytics at the speed of thoughts with HeatWave Lakehouse (by Nitin Kunal)

What is a Data Lake? It’s a cost efficient, scalable, online storage of data as file (for instance, Object Store). Data are not structured and non-transactional and can be sourced from Big Data frameworks.

Again, you could have 4 different platforms to maintain:

Your RDBMS
A DWH system for analytics processing
A Data Lake
A ML & Gen-AI system.

Instead of that, your can merge everything in only one platform: HeatWave. And you can query near real time data with HeatWave Lakehouse because new data is available in seconds: that’s great!

Conclusion

If you have mixed workloads, if you start to work with AI and if you want to improve your performances, it is really worth taking a look at Oracle HeatWave. You can try it here for free.
We all know that AI is the future. These next years, we’ll be more and more challenged on GenAI, ML, vector processing and so on. With all this innovation, we must not lose sight of topics that remain crucial (and perhaps become even more important) such as security, reliability, availability, and best practices. With dbi services and Sequotech we can for sure help you with this transition.

L’article Oracle Technology Roundtable for Digital Natives – Let’s have a look at AI, Cloud and HeatWave est apparu en premier sur dbi Blog.

SQL Server 2022 – Azure Synapse Link

Christophe Cosme — Thu, 15 Sep 2022 08:11:53 +0000

Why is this feature interesting?

The Azure Synapse Link for SQL Server 2022 will enable you to replicate your operational data into Azure Synapse Analytics dedicated SQL pool.

Azure Synapse is an enterprise analytics service that accelerates time to insight across data warehouses and big data systems. Azure Synapse brings together the best of SQL technologies used in enterprise data warehousing, Spark technologies used for big data, Data Explorer for log and time series analytics, Pipelines for data integration and ETL/ELT, and deep integration with other Azure services such as Power BI, CosmosDB, and AzureML.

Diagram of Azure Synapse Analytics architecture.

The advantage of creating a Synapse Link to your SQL server 2022 is that you can replicate your operational data in SQL Server to the Synapse dedicated SQL pool in an automatic way.

The data is first loaded in a landing zone and than copied to the Synapse dedicated SQL pool. The Azure Data Lake Storage Gen2 account is used for the landing zone. This data lake storage must be dedicated only for this Synapse Link for SQL landing zone.

You will be able to use the full power of Synapse to analyze your data without impacting the operational applications. You are also able to federate the data of several SQL Server 2022 databases without having to build complicated load processes.

Prerequisite

On Azure side, you need first to have a Synapse workspace created

On your data center side you need to have SQL Server 2022 Instance installed
- Make sure to have an master encryption key created on the database you want to link with your Synapse Analytics
  - CREATE MASTER KEY ENCRYPTION BY PASSWORD = ‘************’
- Be also sure not to have clustered columnstore indexes on the tables you want to link in Synapse. This is not supported

Create an Azure Data Lake Storage Gen2 account (different from the account created with the Azure Synapse Analytics workspace) used as the landing zone to stage the data submitted by SQL Server 2022.
- If you don’t have a container available, create one that will be used for the landing zone of the data coming from your SQL Server 2022 instance on-premises

You will also need a Share Access Signature (SAS) on your container later on to allow to access it
- In the Azure portal navigate to your storage account you will use for the landing zone
- Select the containers
- Click the ellipse button of the container you created previously
- Select generate SAS

Use User delegation key as the signing method
Select Read/Write/List in the permissions
Set the validity period of your signature (max duration is 7 days, after that you have to renew the signature)
Defined the IP address of the computer which will access the landing zone container
Select the protocol communication protocol
Generate the SAS token
Copy and save the token generated, you will need it later

Make sure that the computer you are working on is allow to access your synapse workspace. Check the firewall rules.

In the Azure portal navigate to your synapse workspace

select the networking option
Enable the checkbox to allow Azure services to access the workspace
add the IP address of your client. Check that it appears in the list of rules
save your settings.

Now that the prerequisites are fulfilled we can start with the real setup of the SQL Server 2022 link with Synapse.

Create your target Synapse dedicated SQL pool

In Azure portal go to the synapse workspace you created and open the synapse studio.

Select the manage hub
Select SQL pool
Click the +New to add a new SQL pool
Key in a name
choose the performance level depending of your needs
create your SQL pool.

It takes some times until the SQL pool is deployed. Check the online indicator

Create a master key for your synapse SQL Pool.

go to data hub
select your SQL pool
create a new SQL script with the empty script option

If you just created your dedicated SQL pool refresh your synapse studio, it might be that the SQL database does not appear in the workspace list.

Paste the command CREATE MASTER KEY in the script pane and run it. Be sure the script is successfully executed.

Create linked service for your source SQL Server 2022

Create a Self-hosted integration Runtime : is a software agent that you can download and install on your on-premises machine or a virtual machine. It is a required component for synapse to get access to the data on SQL Server 2022

The Self-hosted Integration Runtime is only supported on Windows Operating System

Select the [Manage Hub] option
Select the option linked services
select +New
search for “SQL Server ”
select the SQL Server Icon
validate clicking continue.

Define a name for your linked service
Define your owned integration runtime, by opening the dropdown
select +New

Select the Self-Hosted option
validate with continue

Define a name unique in the workspace matching the naming convention
and click create.

I opted for the manual setup. So I downloaded and installed the integration runtime manually. Copy the key in when installing it when required.

When the installation is finished the configuration manager will open. Copy the key provided by the setup in the synapse workspace and click register

Check that the validation is ok and click finish to finalize the registration

Check if the registration is successful

You can now finalize the creation of the linked service on your synapse workspace.

Check that your integration runtime is valid
Capture the name of you on-premises SQL Sever instance
Specify the database name
Define the authentication to your on-premises SQL Server instance.
Test your connection
Click create

The last think you can do for this step is verify that your integration runtime is up and running

Go to the monitor hub
Select integration runtimes
Check the status of your self-hosted integration runtime

Create a linked service to connect to your data lake staging area

First you have to grant permissions to your Azure data lake gen2 for your synapse workspace to access it. On the Azure portal, go to the Azure Data Lake Storage Gen2 account you created.

Select the Access Control (IAM) menu.
Click +Add, and select Add role assignment.

Search for Storage Blob Data Contributor for the selected role
Select the line in the list
Click next

Choose Managed identity in the assign access to section
Click select members

In the right pane

Select the subscription to use
Select Synapse workspaces in the managed identity dowpdown
Select the synapse workspace you want to grant the role to.
Click select

Check if you selected the right member
Validate with Review + assign

This may take a few minutes to take effect to add role assignment.

Now you can create the linked services in your synapse workspace to bind the data lake gen2.

Go back to the synapse studio

Select the manage hub section,
Select linked services
Click +New
In the right pane
Search for Azure Data Lake Storage Gen2
Click continue

Defined a name for your linked service
Select the AutoResolveIntegrationRuntime connection method
Select Accountkey in the authentication type
Select the option Fron Azure subscription
Select the subscription
Select your storage account name to be the destination of your landing zone
Test your connection and check if everything is fine
Finalize the creation by clicking create

Do not to forget to publish the changes when needed

Click the Publishing command
Click the publish button after having reviewed the list of changes to apply

Create the Azure synapse link connection

First be sure that all the tables you need to replicate in your Synapse have a primary key on the on-premises SQL Server instance side. This is mandatory for the data replication

Stay in your Synapse Studio to set this configuration

Go to the integrate hub
Click on the +
Select the Link connection option

Define your new connection to the SQL Server 2022 instance you have on-premises

In the New link connection window

Define the source type, in our case SQL Server
Select the Source linked server you created previously to bind your SQL Server instance on-premises
Select the tables to be replicated on Synapse side
Click continue

In the next step

Select the SQL pool you have created previously
Click continue

Give a name to your link connection
Set the number of core dedicated to the connection
Select the mode of the connection
Sekect the linked service to the landing zone your created previously
Select the storage account container you created (see the prerequisite chapter)
Copy the storage container SSAS token you generated (see the prerequisite chapter)
Click the OK button to create the link connection

On the next screen, you will get a summary of the whole link connection your just defined

Change the target table names, distribution type/Column and structure type to fit best the data you are dealing with
Test again the connection with your SQL Server instance on-premises
Publish the link connection your created on your synapse workspace

Start and control the data replication

Now we are finally ready to start the data replication from your SQL Server 2022 on-premises to your Synapse workspace.

It can take several minutes for the link to take effect and data starting to be replicating.

It will first create the table structure on the SQL pool, than snapshot the data to make an initial data load and start the replication.

In the Synapse Analytics Studio

go to the Integrate hub
Select the link connections you create previously
Click on the Start button

You can follow the status of the data replication from the Synapse Studio

Go to the monitoring hub
Select the Link connection option
Be sure that the status is running
Notice as well that a warning is displayed indicating the landing zone token has an expiring date. I will have to rotate the key to ensure for the replication to continue working
Click on the link connection name to see the detailed information of the replication

Check the status of the replication per table you selected. When error occur you will see it here too. In this case you can click on the error to see the error message.

Control in the SQL pool that the tables have been created correctly and that they contain data

From Synapse Analytics Studio

Select the Data hub
From the Workspace tab, expand the table folder of your SQL pool and check the table created. The specific icon in front of the table indicate that it is a linked table

Hover on the table name
Click the […] button to create action on the table

Select New SQL Script
Click on Select TOP 100 rows to generate the query and see the result set

The query will be written in the SQL script frame
The dataset will be display below

You can query your data and check the replication consistency by changing the query.

You can publish your queries and thus create control script to control if the replication is running as expected.

Check to be done on SQL Server side

There are some checks you can do on your SQL Server side.

Within the database use to link the tables in Synapse, you will find 3 new system tables in which you can query information about the linked tables using the change feed feature

[changefeed].[change_feed_settings]

[changefeed].[change_feed_table_groups]

[changefeed].[change_feed_tables]

You will find connection info used by Synapse to replicate the table through the link service running the following query on your database level:

use []

select * from sys.database_scoped_credentials

And check the result you obtain:

The name of the scoped credential must match your Azure Storage Account container folder your defined
The principal id must match the database user id you used to connect to the database through the Microsoft integration Runtime service

On table level check the propoties. You notice that the Table is replicated is set to True.

Limitation

In the documentation you can find information on the limitations you have at the moment. You will also find the know issues:

https://docs.microsoft.com/en-us/azure/synapse-analytics/synapse-link/synapse-link-for-sql-known-issues

Conclusion

Even if this feature is only in preview mode, and if it seems a bit cumbersome to configure, I like the idea to be able to replicate your operational data to Synapse for real-time data analysis.

It can be a good setup for such scenario.

My next step will be to test how it behaves in a more changing environment, like having lot of transactions on the source system or changing the table structure and so on.

Another aspect I want to experiment, is to plug Power BI reports on this Synapse Analytics linked tables.

It is really a feature I will follow and I’m looking forward to see the further development.

L’article SQL Server 2022 – Azure Synapse Link est apparu en premier sur dbi Blog.

Data Platform Summit 2022 – Analytics at scale with Power BI and Azure Synapse

Christophe Cosme — Wed, 14 Sep 2022 06:45:17 +0000

Introduction

Introduction

This morning I attended the first part of the pre-conf of the Data Platform Virtual Summit 2022 titled “Analytics at scale with Power BI and Azure Synapse” presented by Dave Ruijter.

Working a lot with Power BI and gathering experience with Azure Synapse Analytics , I was looking forward to have Dave’s point of view and experience on this combination of technologies.

It is always good to stay open minded and learn from other expertise and opinion.

We have seen first what are the challenges faced with Power BI and then Dave’s architecture proposal using Azure Synapse Analytics data lake to workaround these challenges.

Power BI Challenges

The 2 main challenges with Power BI is the volume of data that you can handle with it and the second is to show near real-time data.

To bypass the data volume issue you can use the DirectQuery mode instead of importing the data in Power BI. This way you will avoid loading data into Power BI and saving a lot of time to get data accessible in your Power BI report

But the DirecQuery mode has its limitation too. That what starts to be interesting during this session, is how to find solution to minimize these limitations.

The main issue with the DirectQuery mode is the slower performances to gather data and show it to the end user, the report becomes less attractive for the Power BI user experience and the dynamic aspect of the report is fading.

One simple possibility is to apply the query reduction within your report, avoiding to run queries each time the user applies a filter, and you will have to refresh the data manually when you are done with the filter you want to apply.

Azure Synapse Analytics

To increase user experience and bring better performance showing near-real time data with acceptable performance a possible solution is to use Azure Synapse Analytics.

Synapse has been developed a lot since the last couple of years and you have possibility to collect your data from your source system using the API. There are different ways to do it, but the combined use of pipelines and Spark Notebooks is a Dave’s recommendation as they are well integrated into the Synapse Analysis Studio.

Notebooks can be very practical when you have complex scenario to integrate your data. You can developed it using different programming languages within the same notebook and it’s easy to document it within the coding.

Dave’s proposal to to store the data is a Azure Synapse Analytics delta lake using the Bronze, Silver, Gold layers approach. That sound a very good option too.

The delta lake is coming on top of classical data lakes. They provide ACID transactions and will help you to build-up your data sets and history it easily and efficiently.

Bronze layer is used to acquire the original raw data and keep the its history.
Silver layer serves to cleanse, refine, filter and format the data.
Gold layer is finally where you have you will prepare your data to consume it in your reporting system. It must be then fitting your needs by transforming, merging, aggregating, and modelling it, using star schema.

In order to access the data you can use Synapse serverless SQL pools.

For the serverless SQL pools there are also some information to know and best practices to follow:

Serverless SQL pools must be warmed up. After a idle moment it goes in sleeping mode and it will take some time to wake it up again.
Performances can be improved by
- Using the appropriate authentication method
- Locate them in the same region as your Power BI and your end users
- Privilege the parquet file format or data tables as they provide better performance
- Consider storing files with size between 100MG and 10GB to avoid to many costly i/o
- Use appropriate datatype
- Create partition and use filename and filepath function to target them

So your final architecture could be like this one.

And if you still need improvement

Having the solution in place, you can face potential performance issues.

They are some tools to consider to analyze what happen before making changes.

Interesting was the demo using Power BI Performance Analyzer. You can copy the DAX Query directly from the Power BI Performance Analyzer and paste it in the DAX studio for further analysis with the Server Timing option.

VertiPaq Analyzer will give you a good understanding about the data volume you are dealing with in your Power BI report.

With this tools you will be able to know where the bottlenecks are and take action accordingly to optimize the response time of your Power BI report.

After having identified the potential problem you have several possibilities to improve your report:

Make changes in your data model
-Using query folding when possible with your source system
Defining aggregation tables from your detailed info
Adapt the storage mode
Using Hybrid tables
Incremental refresh

Be careful when using the incremental refresh you won’t be able to download the PBIX file from Power BI Service.

So backup your PBIX file versions in safe locations.

Monitoring and Cost control

You can now as well send information from the Power BI to Azure Log Analytics as well, but this feature is still not completely functional

You can find as well a Power BI report template on GitHub to analyze your Power BI platform activities

You can put alert on the log analytics to be warned when some events arise. You can use KQL (Kusto Query Language) to set it up.

It sis recommended to set data consumption limits per day, week and month on your Synapse Analytics. When you use DirectQuery a lot of transactions will run over your SQL serverless pools, they can treat a high volume of data and therefore high costs. This will help you to control your costs on handled by SQL serverless pool.

It is as well recommended to set budget on your subscriptions and resource groups level

Conclusion

It was an interesting session, I could grab some very good information and ideas.

Using delta lake as a data source for Power BI report is something I will definitely test and evaluate.

I could only regret that the demos were not really functioning as expected or not illustrating enough the subject.

But overall it was a great session with lot of valuable information and hints.

L’article Data Platform Summit 2022 – Analytics at scale with Power BI and Azure Synapse est apparu en premier sur dbi Blog.

One day training @ Microsoft Azure Cloud Workshop

Stéphane Haby — Wed, 22 Jan 2020 07:52:32 +0000

Today, with my colleague Christophe, we follow the Microsoft Azure Training Day: Data and Analytics @ Microsoft Zürich (Wallisellen to be exact).

After a presentation of SQL Server 2019, we begin with the First Lab about SQL Server 2019 and new features and we going through intelligent query processing, Data Discovery and Classification for personally identifiable information (PII) and General Data Protection Regulation (GDPR) and secure enclave.

One of my favorites part of the lab was to play/configure a secure enclave because inever do it.
A little tips: Don’t forget to enable parameterization for Always Encrypted in the Query Options

The Second Lab is about Big Data with SQL Server 2019 Big Data Clusters

In this second lab, we work with Azure Data Studio and make some external table with csv file and an external Azure SQL Database.
I was curious to see the execution plan and the cost of these remote queries

Little pause with the lunch and go back to work with our labs!

We begin the afternoon with the database as a Service (DBaaS – PaaS) on Azure: Azure SQL Database (Single, Elastic Pool and Managed Instance) and Cosmos DB the globallly distributed, massively scalable, multi-model database Service.

Now, go to do the Third Lab with Cosmos DB

After somes errors due to my modifications on the alias name of my ressources, it’s finally working good

Little tips: You can choose now Switzerland North as Region

After a little break, the last Lab is about the Modern Data Warehouse and all Azure “BI services” like Azure Data Factory, Azure Databricks, Azure SQL DW and Power BI. The goal of this lab is to learn how these tools can be use together

I discover also a new service in the “Azure Data World”: Azure Synapse
It is Azure SQL Data Warehouse evolved. You can find more information here

At the end, I build my KPI under Power BI very easily…

I will really thanks Microsoft and their Staff for the organization and this useful and interesting workshop day!

A learn a lot and can share with all my customers this experience now

L’article One day training @ Microsoft Azure Cloud Workshop est apparu en premier sur dbi Blog.

Documentum – FT – FTIntegrity Tool

David Diab — Tue, 03 Dec 2019 17:00:55 +0000

FTIntegrity is used to verify indexing after migration or to get a status after a normal indexing. The tool is a standalone Java program that verifies all types that are registered in the dmi_registry_table with the user dm_fulltext_index_user. By default, the utility compares the object ID and i_vstamp between the repository and xPlore.

In this blog I will share with you the default FTIntegrity capabilities, and also some optional configurations.

Default Configuration

Preparation

A script is given to execute the FTIntegrity Tool, but some preparation are needed before the first use:

Verify that the IndexAgent is started
Define the docbase name:
```
export DCTM_DOCBASE_NAME="MYREPO"
```
Navigate to XPLORE_HOME/setup/indexagent/tools
Substitute the repository instance owner password in the script ftintegrity_for_${DCTM_DOCBASE_NAME}.sh, but of course for security reasons it is highly recommanded to don’t put the password into the script, the workaround is to store the password:
```
stty -echo;read MDP;stty echo
```

Update the script to take the password as a parameter:

sed -i 's/password_change_me/$1/g' ftintegrity_for_${DCTM_DOCBASE_NAME}.sh

Execution

Now you can start the FT Integrity tool, usually it takes long time so it is recommended to execute it with nohup to continue running even if the session is disconnected:

nohup ./ftintegrity_for_${DCTM_DOCBASE_NAME}.sh $MDP > ftintegrity_for_${DCTM_DOCBASE_NAME}_$(date +%Y%m%d).log &

Result

Wait the end of the execution, then check files generated under XPLORE_HOME/setup/indexagent/tools:

-rw-r-----. 1 xplore xplore    15640 Dec  3 09:22 ObjectId-indexOnly.txt
-rw-r-----. 1 xplore xplore     7734 Dec  3 09:25 ObjectId-common-version-mismatch.txt
-rw-r-----. 1 xplore xplore  4237982 Dec  3 09:25 ObjectId-common-version-match.txt
-rw-r-----. 1 xplore xplore 20715741 Dec  3 09:25 ObjectId-dctmOnly.txt
-rw-r-----. 1 xplore xplore     1267 Dec  3 09:25 ftintegrity_for_MYREPO_20191203.log

The script generates four reports:

ObjectId-common-version-match.txt
This file contains the object IDs and i_vstamp values of all objects in the index and the repository and having identical i_vstamp values in both places.
ObjectId-common-version-mismatch.txt
This file records all objects in the index and the repository with identical object IDs but nonmatching i_vstamp values. For each object, it records the objectID, i_vstamp value in the repository, and i_vstamp value in the index. The mismatch is on objects that were modified during or after migration.
ObjectId-dctmOnly.txt
This report contains the object IDs and i_vstamp values of objects in the repository but not in the index. The objects in this report could be documents that failed indexing, documents that were filtered out, or new objects generated after migration.
ObjectId-indexOnly.txt
This report contains the object IDs and i_vstamp values of objects in the index but not in the repository.
These objects were removed from the repository during or after migration, before the event has updated the index.

You can resubmit lists (version-mismatch, dctmOnly, and indexOnly) to align the index with the docbase. To do so start the IndexAgent in normal mode, then click Object File and browse to the file.

Optional Configuration

Use filter

As you can see below the dctm Only is huge so it contains a lot of documents:

-rw-r-----. 1 xplore xplore 20715741 Dec  3 09:25 ObjectId-dctmOnly.txt

This is due to the fact that some filters are configured in the Docbase, and avoid indexing some documents, to check these filters execute the below API query:

?,c,select r_object_id,object_name,primary_class from dmc_module where any a_interfaces='com.documentum.fc.indexagent.IDfCustomIndexFilter'

In my case I get the below result:

r_object_id				object_name																			primary_class
----------------		----------------------------------------------------------------------------		----------------------------------------------------------------------------------
0b01e24080000883		com.documentum.services.message.impl.type.MailMessageChildFilter					com.documentum.services.message.impl.type.MailMessageChildFilter
0b01e24080000884		com.documentum.services.message.impl.type.MailMessageChildFilter..J5_D65			com.documentum.services.message.impl.type.MailMessageChildFilter
0b01e24080000bef		com.documentum.server.impl.fulltext.indexagent.filter.defaultCabinetFilterAction	com.documentum.server.impl.fulltext.indexagent.filter.defaultCabinetFilterAction
0b01e24080000bf0		com.documentum.server.impl.fulltext.indexagent.filter.defaultFolderFilterAction		com.documentum.server.impl.fulltext.indexagent.filter.defaultFolderFilterAction
0b01e24080000bf3		com.documentum.server.impl.fulltext.indexagent.filter.defaultTypeFilterAction		com.documentum.server.impl.fulltext.indexagent.filter.defaultTypeFilterAction
(5 rows affected)

By default the FTIntegrity don’t take in account the filters, to include the filters update the script and add “-useFilter T”, use the below command:

sed -i 's//& -useFilter T/' ftintegrity_for_${DCTM_DOCBASE_NAME}.sh

Check specific type

Use the option -checkType to check a specific object type, use the below command to update:

sed -i '/BATCH_SIZE/s/$/ -checkType dm_document/' ftintegrity_for_${DCTM_DOCBASE_NAME}.sh

In this way, only dm_document are verified.

Compare specific metadata

You can compare metadata values which compares object IDs and the specified attributes, to do so use the option -checkMetadata. Be careful, when this option is used you have to add -checkType option, for example, to compare only the a_is_hidden attribute:
Define the list of attributes in a file (one attribute by line):

cat $XPLORE_HOME/setup/indexagent/tools/doclist.txt
a_is_hidden

Then specify the list of metadata to be checked by the script:

...
... $DSEARCH_DOMAIN $OUTPUT_FILEPATH $BATCH_SIZE -useFilter T -checkType dm_document -CheckMetadata XPLORE_HOME/setup/indexagent/tools/doclist.txt

As a result, you will have two new reports:

Object-Metadata-mismatch.txt
Contains all the objects with metadata that has inconsistencies.
Object-Metadata-match.txt
Contains all the objects with metadata that has valid consistencies.

In this way, you can compare really everything! Don’t hesitate to ask or share your experience with this tool

L’article Documentum – FT – FTIntegrity Tool est apparu en premier sur dbi Blog.

Some words about SOUG Day in Lausanne

Oracle Team — Thu, 14 Nov 2019 15:13:13 +0000

Today I participate to SOUG Day which takes place in Lausanne at the “Centre Pluriculturel et social d’Ouchy”.

After a coffee and a welcome speech by Yann Neushaus, Ludovico Caldara and Flora Barriele,

the event starts with 2 global sessions:

A l’heure du serverless, le futur va-t-il aller aux bases de données distribuées?

Franck Pachot makes a comparison between Oracle products (Active Data Guard, RAC, Sharding) and new distributed databases in order to scale-up and scale-out.
Briefly his talk makes reference to:
– Differences among RDBMS, NoSQL and NewSQL according to the CAP Theorem
– Definition and needs for NoSQL and NewSQL
– Definition of services such as Google’s Cloud Spanner, TiDB, CockroachDB, YugabyteDB.

From DBA to Data Engineer – How to survive a career transition?

Kamran Agayev from Azerbaijan speaks about what Big Data in general is and the transition from DBA to Database Engineer.
He addresses several interesting topics:
– Definition of Big Data
– Which are the skills for a Data Engineer, a Data Architect (more complex competences than “just” being a Database Administrator)
– Definition of products like Hadoop, Kafka, NoSQL
After the coffee break, the choice is between 2 different streams. Here some words about the sessions I attend.

Amazing SQL

Laetitia Avrot from EnterpriseDB talks about SQL, which is much more than what we know. SQL is different from other programming languages but it must be treated as one of them. At school we still learn SQL from before 1992, but in 1999 it changed to add relational algebra and data atomicity. PostgreSQL is very close to this standard. Laetitia shows lots of concrete examples of subqueries, common table expressions (CTE), lateral joins (not implemented in MySQL for the moment), anti joins, rollup, window functions, recursive CTE, and also some explanations about key words such as in, values, not in, not exists.

Graph Database

After the lunch, Gianni Ceresa presents property graph databases as combination of vertex (node, properties, ID) and edge (node, ID, label, properties). To start working with Oracle graphs, we can use PGX (Oracle Labs Parallel Graph AnalytiX). The OTN version is better for documentation. Through a demo, Gianni shows how to build a graph using Apache Zeppelin as interpreter and Phyton and Jupyter to visualize it. And then we can also use it to write some data analysis.

5 mistakes that you’re making while presenting your data

Luiza Nowak, a non-IT girl working with IT people (she is a board member of POUG), talks about IT presentations. There are 4 important parts defining them: the content, the story, the speaker performance and visualization.
Here recurrent errors of IT presentations and how to handle them:
1. Lack of data selection – you need to filter your data, to consider who and where you are talking to
2. Too much at once – you need to divide your content, create some suspense and put less information into slides to let the audience listen to you instead of reading them
3. Forget about contrast – you have to use contrast on purpose because it can be useful to underline something but it can also distract your audience
4. Wrong type of charts – you have to be clear about your data and explain results
5. You don’t have any story – you need to conceptualize your data.

How can Docker help a MariaDB cluster for disaster/recovery

Saïd Mendi from dbi services explains what a MariaDB Galera Cluster is and his benefits and how Docker can help in some critical situations. Actually you can create some delayed slaves which can be useful to emulate the flashback functionality.

Conclusion

SOUG Day arrives to an end. It was a nice opportunity to meet international speakers, discuss with some customers and colleagues, learn and share. As usual, this is part of dbi services spirit and matches with our values!
And now, I have to say you goodbye: it’s the aperitif and dinner time with the community Hope to see you at the next SOUG event.

L’article Some words about SOUG Day in Lausanne est apparu en premier sur dbi Blog.

Pass summit – day 4

Christophe Cosme — Fri, 09 Nov 2018 17:01:34 +0000

Optimizing Multi-Billion Row Tables in Tabular in 2018

I wanted to attend the session moderated by Marco Russo to see his approach of optimizing performance in Tabular model
The first thing to understand is how the data is stored and organized in a Tabular model. It is of course using the xVelocity in-memory capabilities with the Vertipac column storage layout engine. It organizes the data, compressing it by column, in combination with a dictionary encoding mechanism.
Therefore the most important aspects in a model to take care at are:

Reducing the number of distinct values of columns
Focus on columns with many distinct values

What impacts the performance of data retrieval is also the relationships in the model that define the cardinality. The higher the cardinality is, the slower the query will be.

In order to measure there criteria the following tools are available:

Vertipac analyzer: it can help you to get insight on your tabular model, to get information on the number of items in your columns and their size.
DAX studio: it is an open source that you can download visiting https://Daxstudio.org

DAX Studio will help you to analyze how your queries are performing and you will be able to see how the performance is related to the number of cores, the size of the columns queried and as already said above the size of the cardinality involved.

What is recommended for tabular model is to have the fastest CPU speed (>= 3GHz) and CPU L2/L3 large caches; large cache of the CPU is essential to avoid quickly performance problems, but having to many CPU can be counterproductive, having more sockets can negatively impact your performance. Fastest RAM is also recommended(>=3200). You have to be aware of the latency of memory and storage access and see the huge impact it has on the performance.

Marco delivers best practice using the Azure tiers available

The way you can influence having too high cardinality between fact and dimension is reduce the dimension with sub category but also to reduce you fact by splitting it and doing aggregates. You have to re-shape your data model.

The great thting is that a new “Dual storage mode and aggregation option” is a new feature in Power BI Premium that enable to defined aggregation table of your big fact tables that do not fit in memory within Power BI.

Again an interesting data analytic day…stay tuned..

L’article Pass summit – day 4 est apparu en premier sur dbi Blog.

Pass summit – dbi visit day 3

Christophe Cosme — Thu, 08 Nov 2018 15:52:38 +0000

The Microsoft data platform is evolving

The third day began with the Keynote help by Rohan Kumar the Corporate Vice President of Azure Data by Microsoft

The main message was “Hybrid data platform is the way for the future” and Microsoft is working in this direction.

The use of AI and analytic to transform the customer business is also a key driver for the Microsoft data platform, which is building to enable this easily.

Rohan remember the customer priorities:
– Modernizing on-premises
– Modernizing the cloud
– Building cloud native cloud application
– Unlocking insights

To support these priorities we got an overview of the last development on the data platform product or services:
Modernizing the on-premises solution with SQL Server 2019
○ More language support
○ Best security and performance
○ Power BI Report Server
○ Mission critical availability and performance
○ Security and compliance
○ Management and development
○ Big data and analytics

And Microsoft is announcing SQL Server 2019 CPT 2.1

One big evolution in term of performance is that some system tables will be moved in in-memory technology.
SQL Server 2019 is becoming a hub for integrating data and also enable the data injection in big data clusters combining Spark and SQL Server

Azure Data Studio si the new tool to manage all the Azure data services in order to see them a unified data platform

Modernizing the cloud with the new services or their new capabilities or functionalities
Azure database migration service
Azure SQL database managed instance with a business critical option available
Azure SQL Database Hyper scale (up to 100TB)
Accelerated databases recovery and machine Learning services in Azure SQL Database
Building cloud-native apps with Azure Cosmos DB
Multi-Master and Cassandra API

Unlocking insights
New features of Azure SQL Data Warehouse, like workload management
Partnership Informatica, tableau connectors
HDInsight 4.0
Real-time analytics
Azure event hubs for Kafka
Azure Data Explorer
Power BI Dataflow
Power BI SSRS Reports in Power BI
Azure Databricks new feature

The SQL Server product is evolving constantly and the Azure world event faster.

Again an interesting data analytic day…stay tuned..

L’article Pass summit – dbi visit day 3 est apparu en premier sur dbi Blog.

Pass summit – dbi visit day 1

Christophe Cosme — Mon, 05 Nov 2018 23:16:53 +0000

Designing Modern Data and Analytic Solution in Azure

After having explained the pros and cons of Azure and the decision drivers for going to an Azure Architecture, some interesting messages were delivered like the decoupling of the storage and compute aspect on Azure, even if some of services still combine both. Another message that we all know but is essential to remind on a regular basis is that cost control is an important aspect. Developers and architects can influence the cost of cloud based platform hugely.

The fact that Azure is evolving a lot, that new services are offered or that their features are changing , you have to be constantly informed about these new or deprecated capabilities. You are sometimes faced to some migration works to adapt to these evolutions. Proof of concept must be an habit, a routine due to this constantly moving environment.

Organizing and configuring your resources in Azure is also something to consider seriously not to be in trouble and lost with all the resources you deployed within your organization. You have some notions that become important to use to organize your Azure platform like

Subscriptions
Resource groups
Tags
Policies
Locations of your services
Naming convention

Using Tags on Azure can help a lot for reporting reasons.
A right naming convention (i.e. Purpose + type of service + environment), can help you to deploy easier your different environments, but a trick is also not use special characters because they cannot be used consistently in azure services. Using policies will support you to control the services are created the way your governance wants them.

A presentation of a data analytic architecture was presented including the following services in order to managed structured data:

Azure data factory
Azure Blob Storage
Azure data bricks
Azure SQL Data Warehouse
Azure Analysis Services
Power BI

Then following services are suggested additionally to add real-time data analytic

Azure HDInsight Kafka
Azure Databricks Spark
Azure Cosmos DB

They presented then the different layers of the architecture and the possible azure services, data acquisition, data storage, data computing and data visualization.
When doing on premise data warehouse project for 15 years , I was mentioning always that I want to privilege ELT vs. ETL , I was glad to hear that now ELT is really mostly used terminology and Data Factory was presented that way. The new version enables now to run SSIS packages too. The different data storage possibilities, like blob storage, data lake, SQL DWH, SQL DB and Cosmos DB have been reviewed. Choosing the right data storage is always to find the right balanced between data schema read and data schema write. But in a modern BI Azure platform you will find a a so called polyglot data storage solution, combining several types a storage services.

A component often emerging in this tools constellation is Databricks running Apache Spark for the data engineering and data science. The advantage is that it is closer to open source world and support several languages SQL, Python, R, Scala and can be used for use cases like bath data processing, Interactive analytics, machine learnings, Stream event processing,…

In such environment, the data virtualization is also an interesting capabilities on such architecture. It is possible using for instance polybase, allowing to query the disparate stored data avoiding to replicate, duplicate it.

The per-conference day finished with the presentation of Power BI and Azure Analysis Service and some automation technics with ARM and PowerShell

Again an interesting data analytic day…stay tuned…

L’article Pass summit – dbi visit day 1 est apparu en premier sur dbi Blog.