Yesterday I participated to the Oracle Technology Roundtable for Digital Natives in Zurich.

It was a good opportunity to learn more about AI, Cloud and HeatWave with the focus on very trendy features of this product: generative AI, machine learning, vector processing, analytics and transaction processing across data in Data Lake and MySQL databases.

It was also great to share moments with the Oracle and MySQL teams and meet customers which gave feedback and tips about their solutions already in place in this area.

I’ll try to summarize below some key take-away of each session.

Unlocking Innovation: How Oracle AI is Shaping the Future of Business (by Jürgen Wirtgen)

AI is not a new topic. But how do we use it today and where are we in the process, early or advanced?

To answer this question, you can have a look to the stages of adoption:

1. Consume (AI embedded in your applications) -> SaaS applications
2. Extend (models via Data Retrieval, RAG) -> AI services
3. Fine tune -> Data
4. Build models from scratch -> Infrastructure

AI is not AI. The best AI starts with the best data, securely managed. Which can be translated back into a simple equation: Best Data + Best Technology = Best AI.

Innovations in HeatWave & MySQL – The Present and the Future (by Cagri Balkesen)

HeatWave is an in-memory query processing accelerator for data in MySQL transactional RDBMS or data in Object Store in different format.

Normally you need to put in place and maintain ETL processes to produce data that can be used effectively by analytics. This brings several drawbacks:

• Complexity
• You have to maintain different systems, on which you’ll have security issues to handle and costs to assume.

Using HeatWave, you don’t need that anymore, because it’s a single platform which allows you to manage together all your OLTP, OLAP, Machine Learning and GenAI workloads.

Which are the advantages of using HeatWave?

• You current SQL syntax doesn’t need to change
• Changes to data are automatically propagated to HeatWave
• Best performances for your queries
• Efficient processing for Data Lake
• Best platform for MySQL workloads
• Built-in GenAI & Vector Store
• Available in multi cloud (natively on OCI, it can run inside AWS, you can setup a private interconnection for Microsoft Azure, and there are works in progress for Google Cloud).

HeatWave is based on a massively parallel architecture which uses partitioning of data: each CPU core within a node processes the partitioned data in parallel.

Driven by Machine Learning algorithms, HeatWave Autopilot offers several features such as:

• Improvements in terms of performance and scalability
• Provisioning, data loading, query execution and fault management automation, to reduce human errors.

Finally, according to Oracle, with HeatWave you will have best performances and lowest costs rather than competitors: Snowflake, Amazon Redshift, Google BigQuery and Databricks.

Building Next-Gen Applications with Generative AI & Vector Store (by Adi Hochmann)

As we said, Oracle HeatWave allows you to manage together all your OLTP, OLAP, Machine Learning and GenAI workloads.

Steps to build a GenAI application are the following ones:

1. Create a vector store
2. Use vector store with LLMs

And this is performed using the following routines:

call sys.HEATWAVE_LOAD (…);

call sys.ML_RAG(@query,@output,@options);

But how to train data for Machine Learning? The process and tasks done by a data analyst could be complex and this is replaced here by the AutoML:

CALL sys.ML_TRAIN(‘data_source’, ‘model_type’, JSON_OBJECT(‘task’, ‘classification’), @result_model);

This is useful for some use-cases such as classification, anomalies detection, recommendations, predictive maintenance, …

Additional tip: Adi used MySQL Shell for VS Code to run his demo. This extension enables interactive editing and execution of SQL for MySQL databases and MySQL Database Service. It integrates the MySQL shell directly into VS Code development workflows and it’s pretty nice!

Oracle Cloud for Digital Natives: Supporting Innovation and Growth (by Claire Binder)

Which are the 5 reasons why Digital Natives picked OCI?

1. Developer-First openness and flexibility, to speed acquisition
2. Advanced Data & AI Services, to achieve innovation and agility
3. Technical and global reach, to achieve scalability
4. Security, compliance and resilience, to control risks
5. Cost efficiency and TCO, to achieve optimized spending.

Linked to the point 4, there are several services in OCI to avoid data breaches in terms of prevention, monitoring, mitigation, protection, encryption and access.  

To select a Cloud provider, the recommendation would be to choose a solution which allows you to run converged open SQL databases, instead of single-use proprietary databases.

And finally, Oracle brings AI to your data with his new 23ai release and some of its features, such as Property Graphs and AI Vector Search.

Analytics at the speed of thoughts with HeatWave Lakehouse (by Nitin Kunal)

What is a Data Lake? It’s a cost efficient, scalable, online storage of data as file (for instance, Object Store). Data are not structured and non-transactional and can be sourced from Big Data frameworks.

Again, you could have 4 different platforms to maintain:

•  Your RDBMS
• A DWH system for analytics processing
• A Data Lake
• A ML & Gen-AI system.

Instead of that, your can merge everything in only one platform: HeatWave. And you can query near real time data with HeatWave Lakehouse because new data is available in seconds: that’s great!

Conclusion

1. If you have mixed workloads, if you start to work with AI and if you want to improve your performances, it is really worth taking a look at Oracle HeatWave. You can try it here for free.
2. We all know that AI is the future. These next years, we’ll be more and more challenged on GenAI, ML, vector processing and so on. With all this innovation, we must not lose sight of topics that remain crucial (and perhaps become even more important) such as security, reliability, availability, and best practices. With dbi services and Sequotech we can for sure help you with this transition.

L’article Oracle Technology Roundtable for Digital Natives – Let’s have a look at AI, Cloud and HeatWave est apparu en premier sur dbi Blog.

There is now CNCF-hosted Co-located events adding more topics and interesting sessions. This is community-driven, vendor-neutral events hosted and managed by the CNCF. ArgoCD, Cilium, Linkerd have now their own conference. Here is the list of CNCF-hosted Co-located events.

Unfortunately, all the above events were already sold out when we decided to come at the KubeCon.

During the 1^st day Keynote, Chris Aniszczyk CTO of the CNCF mentioned that for this year, the increase of the number of participants was just amazing reaching more than 10’000 people. The number of CNCF projects also increased a lot. I feel that the main message of this Keynote is that there is a real need for contributors to help maintainers – more projects but seems less contributors.

The keynote was interesting, I learned that there are new certifications available in the CNFC cert path.

I also learned that GitLab choose flux for the GitOps capabilities. It’s already documented in the GitLab documentation here. See as well this blog announcement.

I noticed a project that I would like to follow: The scalable, reliable, MySQL-compatible, cloud-native database. I’m quite sure my database colleagues Elisa Usai & Saïd Mendi would love to do a presentation during our next dbi xChange event. I would encourage them to have a look at vitess.io.

After the keynotes I participated to the following sessions

• Learn the Helm Code Base and PR Review Process
• Emissary-Ingress: Self-Service APIs and the Kubernetes Gateway API
• Argo CD Core – A Pure GitOps Agent for Kubernetes
• How to Turn Release Management from Duty to Fun: Lessons Learned Building the Cluster API Release Team
• Highly Available Routing with Multi Cluster Gateways

Following some information about the sessions.

Learn the Helm Code Base and PR Review Process.

Scott Rigby, Andrew Block & Karena Angell were holding the session and explained rapidly what helm is. They focus the session on drawing attention to the need for contributors. Anybody could help not only by developing but also in doing review, documentation, bugs triage and other stuff that may not require high level of expertise. We explored the git repository with the composed directory explaining each of them. How the source code of the helm tool is composed. It was interesting but to be honest, I was thinking that we would have to describe which new features are in the pipe or other cool stuff around the futur of helm. It seems that maintainer are already overloaded and except maintaining the code there is no real new features or at least it was not covered during this session.

Emissary-Ingress: Self-Service APIs and the Kubernetes Gateway API

Lance Austin and Flynn Buoyant provides update on the emissary-ingress, including new features since the time it was introduced in Detroit. They also explained the needs of having a self-service, developer centric configuration tools for APIs.

Argo CD Core – A Pure GitOps Agent for Kubernetes

The speakers, specially the co-creator of Argo provides description of the Argo CD component architecture, showing the internal layer saying that the core is decoupled from the UI. The core can be easily used by Kubernetes Administrator to deploy several applications along to multi-cluster. A demonstration was done to show how Kube Admin could use Argo CD as a pure GitOps Agent.

I think you now have an overview of the session I’ve followed today, it’s a complete program during the KubeCon & CloudNativeCon. I had to choose between several streams as there were interesting other sessions in parallel. I’m really happy to be part of this event but now it’s time to take a bier with my colleague Benoît and see what will do tonight – visiting Amsterdam most probably.

L’article The 1st Day at KubeCon & CloudNativeCon 2023 in Amsterdam est apparu en premier sur dbi Blog.

During a consulting on a customer, we faced the following issue when trying to delete a dbhome

Message:  DCS-10001:Internal error encountered: PRGO-2470 : Working copy “OraDB19000_home1” is involved in an incomplete move or upgrade operation

The result of the job is shown below

[2022-11-08 10:53:35 root@odaserverb]# odacli describe-job -i 671b5899-02ac-45ff-b5af-ee254ef0bc72

Job details
----------------------------------------------------------------
                     ID:  671b5899-02ac-45ff-b5af-ee254ef0bc72
            Description:  Database Home OraDB19000_home1 Deletion with id a1bfe23e-2569-407b-8b87-7af9f9f586bf
                 Status:  Failure
                Created:  October 26, 2022 6:45:49 AM CEST
                Message:  DCS-10001:Internal error encountered: PRGO-2470 : Working copy "OraDB19000_home1" is involved in an incomplete move or upgrade operation..

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
DbHome service deletion for a1bfe23e-2569-407b-8b87-7af9f9f586bf October 26, 2022 6:45:49 AM CEST    October 26, 2022 6:46:02 AM CEST    Failure
DbHome service deletion for a1bfe23e-2569-407b-8b87-7af9f9f586bf October 26, 2022 6:45:49 AM CEST    October 26, 2022 6:46:02 AM CEST    Failure
Validate dbhome a1bfe23e-2569-407b-8b87-7af9f9f586bf for deletion October 26, 2022 6:45:50 AM CEST    October 26, 2022 6:45:50 AM CEST    Success
Setting up ssh equivalance               October 26, 2022 6:45:52 AM CEST    October 26, 2022 6:45:55 AM CEST    Success
Setting up ssh equivalance               October 26, 2022 6:45:55 AM CEST    October 26, 2022 6:45:58 AM CEST    Success
Deleting DbHome by RHP                   October 26, 2022 6:45:58 AM CEST    October 26, 2022 6:46:02 AM CEST    Failure

After researches on the net, we found following Oracle document

ODA: Create-prepatchreport fails with PRGP-1005 – cannot specify databases (Doc ID 2900263.1)

This document is not about the same issue, but we can find inside that the table MOVE in the database MySQL should be empty in a normal state.

According this document if this table contains rows, it indicates that a move have failed in the past. Indeed we did an upgrade dbhome which failed.

So we decided to follow the steps described in this document and then retry the delete dbhome.

We just want to inform that we do these steps at our own risks and that it’s up to you to execute or not following commands  without Oracle support

As we are using an HA ODA, we do following tasks on both nodes

We first connect to the MySQL Metadata repository

[2022-11-08 10:54:07 root@odaserverb]# cd /opt/oracle/dcs/mysql/bin
[2022-11-08 11:31:58 root@odaserverb]# ./mysql -u root --socket=/opt/oracle/dcs/mysql/log/mysqldb.sock

We then locate on wich database is the table MOVE

mysql> select table_schema as database_name, table_name
    -> from information_schema.tables
    -> where table_type = 'BASE TABLE'
    ->     and lower(table_name) like '%move%'
    -> order by table_schema,
    ->     table_name;
+---------------+------------+
| database_name | TABLE_NAME |
+---------------+------------+
| GHSUSER23     | MOVE       |
+---------------+------------+
1 row in set (0.00 sec)

Let’s connect to the GHSUSER23 database and let’s query the table MOVE. And we can see the table is not empty.

mysql>use GHSUSER23;

mysql> select NAME,SRCWC,SRCHOME,DSTWC,DSTHOME from MOVE;
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------+------------------------------------------------------+------------------+------------------------------------------------------+
| NAME                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         | SRCWC            | SRCHOME                                              | DSTWC            | DSTHOME                                              |
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------+------------------------------------------------------+------------------+------------------------------------------------------+
| 0xACED0005737200116A6176612E7574696C2E48617368536574BA44859596B8B7340300007870770C000000103F400000000000017400084F4643434D50524578                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           | OraDB19000_home1 | /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_1 | OraDB19000_home2 | /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_2 |
| 0xACED0005737200116A6176612E7574696C2E48617368536574BA44859596B8B7340300007870770C000000023F400000000000017401354F4643434D5052453B4F7261444231393030305F686F6D65313B2F7530312F6170702F6F64616F7261686F6D652F6F7261636C652F70726F647563742F31392E302E302E302F6462686F6D655F313B4F7261444231393030305F686F6D65323B2F7530312F6170702F6F64616F7261686F6D652F6F7261636C652F70726F647563742F31392E302E302E302F6462686F6D655F323B6F7261636C653B31392E302E302E302E303B5241433B747275653B4E4F545F5350454349464945443B4E4F545F5350454349464945443B66616C73653B4E4F545F5350454349464945443B66616C73653B66616C73653B69726973646576707265613A44425F504F53545F55534552414354494F4E535F535543434553532B69726973646576707265623A53544152545F534552564943455F5355434345535378 | OraDB19000_home1 | /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_1 | OraDB19000_home2 | /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_2 |
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------+------------------------------------------------------+------------------+------------------------------------------------------+
2 rows in set (0.00 sec)

We  decided to empty the table MOVE, but we did a copy before

mysql> create table MOVE_BCK as select * from MOVE;

mysql> select NAME,SRCWC,SRCHOME,DSTWC,DSTHOME from MOVE_BCK;
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------+------------------------------------------------------+------------------+------------------------------------------------------+
| NAME                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         | SRCWC            | SRCHOME                                              | DSTWC            | DSTHOME                                              |
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------+------------------------------------------------------+------------------+------------------------------------------------------+
| 0xACED0005737200116A6176612E7574696C2E48617368536574BA44859596B8B7340300007870770C000000103F400000000000017400084F4643434D50524578                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           | OraDB19000_home1 | /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_1 | OraDB19000_home2 | /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_2 |
| 0xACED0005737200116A6176612E7574696C2E48617368536574BA44859596B8B7340300007870770C000000023F400000000000017401354F4643434D5052453B4F7261444231393030305F686F6D65313B2F7530312F6170702F6F64616F7261686F6D652F6F7261636C652F70726F647563742F31392E302E302E302F6462686F6D655F313B4F7261444231393030305F686F6D65323B2F7530312F6170702F6F64616F7261686F6D652F6F7261636C652F70726F647563742F31392E302E302E302F6462686F6D655F323B6F7261636C653B31392E302E302E302E303B5241433B747275653B4E4F545F5350454349464945443B4E4F545F5350454349464945443B66616C73653B4E4F545F5350454349464945443B66616C73653B66616C73653B69726973646576707265613A44425F504F53545F55534552414354494F4E535F535543434553532B69726973646576707265623A53544152545F534552564943455F5355434345535378 | OraDB19000_home1 | /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_1 | OraDB19000_home2 | /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_2 |
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------+------------------------------------------------------+------------------+------------------------------------------------------+
2 rows in set (0.00 sec)

And then delete the rows

mysql> delete from MOVE;
Query OK, 2 rows affected (0.00 sec)

mysql> commit;
Query OK, 0 rows affected (0.00 sec)

We then retried the delete dbhome

[2022-11-08 12:05:19 root@odaservera]# odacli list-dbhomes;

ID                                       Name                 DB Version                               Home Location                                 Status
---------------------------------------- -------------------- ---------------------------------------- --------------------------------------------- ----------
a1bfe23e-2569-407b-8b87-7af9f9f586bf     OraDB19000_home1     19.14.0.0.220118                         /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_1 FAILED
06d87fc9-7ebb-4944-aefa-a18f82100506     OraDB19000_home2     19.16.0.0.220719                         /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_2 CONFIGURED

And we run the delete-dbhome command on one node

[2022-11-08 12:05:25 root@odaservera]# odacli delete-dbhome -i a1bfe23e-2569-407b-8b87-7af9f9f586bfAnd the

A few minutes after the job finished successfully

[2022-11-08 12:07:33 root@odaservera]# odacli describe-job -i 79ec29f2-24c8-4bc1-a674-695e81127f0a

Job details
----------------------------------------------------------------
                     ID:  79ec29f2-24c8-4bc1-a674-695e81127f0a
            Description:  Database Home OraDB19000_home1 Deletion with id a1bfe23e-2569-407b-8b87-7af9f9f586bf
                 Status:  Success
                Created:  November 8, 2022 12:05:41 PM CET
                Message:

Task Name                                Start Time                          End Time                            Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
Validate dbhome a1bfe23e-2569-407b-8b87-7af9f9f586bf for deletion November 8, 2022 12:05:41 PM CET    November 8, 2022 12:05:41 PM CET    Success
Setting up ssh equivalance               November 8, 2022 12:05:43 PM CET    November 8, 2022 12:05:46 PM CET    Success
Setting up ssh equivalance               November 8, 2022 12:05:46 PM CET    November 8, 2022 12:05:49 PM CET    Success
Deleting DbHome by RHP                   November 8, 2022 12:05:49 PM CET    November 8, 2022 12:07:53 PM CET    Success

And we validate with the list-dbhomes

[2022-11-08 12:07:57 root@odaservera]# odacli list-dbhomes;

ID                                       Name                 DB Version                               Home Location                                 Status
---------------------------------------- -------------------- ---------------------------------------- --------------------------------------------- ----------
06d87fc9-7ebb-4944-aefa-a18f82100506     OraDB19000_home2     19.16.0.0.220719                         /u01/app/odaorahome/oracle/product/19.0.0.0/dbhome_2 CONFIGURED

Conclusion

Hope this blog help

L’article ODA : Do You Know The MOVE Table In MySQL DB Repository est apparu en premier sur dbi Blog.

Introduction

This blog is the third chapter related to deploying a MySQL infrastructure on the Azure cloud. In addition to performance, we should indeed also consider backup and restore capabilities. The objective of this blog is to present the main backup and restore possibilities offered by Azure through a simple example and show a second backup/restore possibility using MySQL Shell dump utilities.

Backup window and mechanism

Flexible Azure for MySQL generates by default a 7 days server backup retention period. This retention period can be extended up to 35 days or shorten to 1 day. In addition we can decide if you want to have a Geo Redundant backup storage. By default the backups are locally redundant.

It’s important to understand that Azure makes backup of the whole server, not only the MySQL Server through mysqldump, MySQL Enterprise Backup or any other solution. These backups can only be used to restore MySQL Server in another Azure Database for MySQL Server. It means that these backups cannot be exported to generate a new database on our on-premise server for instance. If we want to extract part of our database in order to export data, we can use mysqldump, MySQL Shell’s instance dump utility or the set of tools provided by MySQL Shell.

The backups provided by Azure can be used to make Point In Time Recovery of the Server with a granularity of 5 minutes since the system snapshots are done automatically every 5 minutes. As specified in the documentation, the backups are encrypted using AES 256-bit.

Backup and restore costs

As explained on Microsoft website

“Backup storage is the storage associated with automated backups of your server. Increasing your backup retention period increases the backup storage that is consumed by your server. There is no additional charge for backup storage for up to 100% of your total provisioned server storage. Additional consumption of backup storage will be charged in GB/month.” – https://azure.microsoft.com/en-us/pricing/details/mysql/flexible-server/

Increasing the retention of the backup, may have an impact on the pricing. We can have an idea of the global costs related to Azure Database for MySQL on the following URL: https://azure.microsoft.com/en-us/pricing/details/mysql/flexible-server/

Recover a database from Azure Backup Restore interface

In this first test, we will simply use the recovery functionalities provided by Azure Database for MySQL flexible server. We will simulate a user error by deleting a table and we will restore the entire server (as it’s not possible to simply recover a database or a table using Azure features).

1. Dropping a table by mistake

 MySQL  albatroz.mysql.database.azure.com:3306 ssl  SQL > SELECT CURRENT_TIMESTAMP ;
+---------------------+
| CURRENT_TIMESTAMP   |
+---------------------+
| 2022-08-18 20:48:44 |
+---------------------+
1 row in set (0.1002 sec)
 MySQL  albatroz.mysql.database.azure.com:3306 ssl  SQL > drop table sysbench.sbtest1;
Query OK, 0 rows affected (0.1884 sec)

2. Restoring the MySQL Server to a time before the mistake

We first have to go in the “backup/restore” menu of our Azure Database for MySQL flexible server and select the backupset that we want to restore. As we can see a backupset is done everyday. In our current context, we want to use the most recent backupset (Automated backup #5)

Once the backupset selected a screen appears showing the restore server options. It provides us with the possibility to make a Point In Time Restore (PITR) of our server by choosing between 3 options:

• Latest restore point (Now)
• Select a custom restore point
• Select fastest restore point (Restore using full backup)

In our case we will use the “Select a Custom Restore point” option as shown in the screenshot below. We will define the custom restore time just before the mistake and specify a name for the restored Server.

Once the restore requested, it took approximately 5 minutes to deploy the new server.

3. Let’s check if my table is back on the new deployed server

Finally we simply have to connect to the new restored server and check if the dropped table is back. Of course we can also export this table from this restored server and import this same table on the original server using mysqldump.

 MySQL  albatrozrestored.mysql.database.azure.com:3306 ssl  sysbench  SQL > show tables from sysbench like '%1';
+-------------------------+
| Tables_in_sysbench (%1) |
+-------------------------+
| sbtest1                 |
+-------------------------+
1 row in set (0.1048 sec)

4. Export/Import table from the restored Server

Now that the server is restored, we can export the table that was deleted by mistake using util.dumpTables() and import it to the albatroz server using util.loadDump(). The process is rather simple as you can see below:

Export from the recovered server (albatrozrestored)

MySQL  albatrozrestored.mysql.database.azure.com:3306 ssl  sysbench  JS > util.dumpTables("sysbench", [ "sbtest1"], "C:/Users/grs/Albatroz-Sysbench-sbtest1");
NOTE: Backup lock is not available to the account 'grs'@'%' and DDL changes will not be blocked. The dump may fail with an error if schema changes are made while dumping.
Acquiring global read lock
Global read lock acquired
Initializing - done
...
...
109% (15.29K rows / ~13.98K rows), 0.00 rows/s, 0.00 B/s uncompressed, 0.00 B/s compressed
Dump duration: 00:00:01s
Total duration: 00:00:06s
Schemas dumped: 1
Tables dumped: 1
Uncompressed data size: 2.93 MB
Compressed data size: 1.33 MB
Compression ratio: 2.2
Rows written: 15294
Bytes written: 1.33 MB
Average uncompressed throughput: 2.35 MB/s
Average compressed throughput: 1.07 MB/s

Import on Albatroz server

MySQL  albatroz.mysql.database.azure.com:3306 ssl  JS > util.loadDump("C:/Users/grs/Albatroz-Sysbench-sbtest1", {schema: "sysbench"});
Loading DDL and Data from 'C:/Users/grs/Albatroz-Sysbench-sbtest1' using 4 threads.
Opening dump...
Target is MySQL 8.0.28. Dump was produced from MySQL 8.0.28
Scanning metadata - done
Checking for pre-existing objects...
Executing common preamble SQL
Executing DDL - done
Executing view DDL - done
Starting data load
1 thds loading | 100% (2.93 MB / 2.93 MB), 1.94 MB/s, 0 / 1 tables done
Executing common postamble SQL
Recreating indexes - done
1 chunks (15.29K rows, 2.93 MB) for 1 tables in 1 schemas were loaded in 6 sec (avg throughput 1.94 MB/s)
0 warnings were reported during the load.

Recover a database from your own backups

As stated in the introduction, I will present in this chapter a complementary solution of backup/restore. Of course Azure does not prevent us from doing our own backups by connecting to the Azure Database for MySQL flexible server and using either mysqdump, MySQL Enterprise Backup or any other MySQL backup solution. I decided to use the opportunity of this blog to use the backup tools provided by MySQL Shell. Indeed MySQL Shell’s instance dump utility, such as util.dumpInstance(), util.dumpSchemas() or even util.dumpTables(), introduced in MySQL Shell 8.0.22, provide interesting functionalities. This export tools alone would deserve several blogs dedicated to them.

Before starting let’s illustrate what will be demonstrated in the next few lines:

1. The very first step consist in doing a dump of the MySQL Instance
2. Secondly, we will insert a row in a table named sbtest1
3. Thirdly, we will simulate a human error and drop a table
4. Then we will restore the database to the state after the backup
5. After having restored the backup, we will execute the binary to the state just before the human error
6. Finally we will check that the last insert I did is stored into the sbtest2

1. Dump of MySQL Instance using util.dumpInstance()

As explained above, the very first step consist in doing a dump of the entire instance. Without this backup we won’t be able to restore the database. We will use util.dumpInstance() as presented below:

MySQL  albatroz.mysql.database.azure.com:3306 ssl  JS > util.dumpInstance("C:/Users/grs/AlbatrozDump", {dryRun: false, showProgress: true, threads: 2})
NOTE: Backup lock is not available to the account 'grs'@'%' and DDL changes will not be blocked. The dump may fail with an error if schema changes are made while dumping.
Acquiring global read lock
Global read lock acquired
Initializing - done
2 out of 6 schemas will be dumped and within them 9 tables, 0 views.
4 out of 7 users will be dumped.
...
...
107% (137.28K rows / ~128.10K rows), 14.42K rows/s, 2.61 MB/s uncompressed, 1.20 MB/s compressed
Dump duration: 00:00:15s
Total duration: 00:00:21s
Schemas dumped: 2
Tables dumped: 9
Uncompressed data size: 26.26 MB
Compressed data size: 11.97 MB
Compression ratio: 2.2
Rows written: 137284
Bytes written: 11.97 MB
Average uncompressed throughput: 1.72 MB/s
Average compressed throughput: 784.25 KB/s

2. Inserting row in our table

Now we simulate some activity in the database by inserting a row into the table sbtest1.

MySQL  albatroz.mysql.database.azure.com:3306 ssl  sysbench  SQL > insert into sbtest1 values(999999999,1,1,"my row before drop table");
Query OK, 1 row affected (0.1134 sec)
MySQL  albatroz.mysql.database.azure.com:3306 ssl  sysbench  SQL > SELECT CURRENT_TIMESTAMP ;
+---------------------+
| CURRENT_TIMESTAMP   |
+---------------------+
| 2022-08-19 16:42:12 |
+---------------------+

3. Dropping a table by mistake

Thirdly, we simulate the human error by dropping the table sbtest1.

 MySQL  albatroz.mysql.database.azure.com:3306 ssl  sysbench  SQL > SELECT CURRENT_TIMESTAMP ;
+---------------------+
| CURRENT_TIMESTAMP   |
+---------------------+
| 2022-08-19 16:46:47 |
+---------------------+
1 row in set (0.1095 sec)
 MySQL  albatroz.mysql.database.azure.com:3306 ssl  sysbench  SQL > drop table sysbench.sbtest1;
Query OK, 0 rows affected (0.1823 sec)

4. Restoring the MySQL Server using the backup

Now, we have to restore the database using the last backup we have. We will use util.loadDump() in order to restore our table. To only recover the table we can simply use the option “includeTables“.

 MySQL  albatroz.mysql.database.azure.com:3306 ssl  sysbench  JS > util.loadDump("C:/Users/grs/AlbatrozDump", { includeTables: ["sysbench.sbtest1"],loadDdl:true, LoadData:true, threads: 2})
Loading DDL and Data from 'C:/Users/grs/AlbatrozDump' using 2 threads.
Opening dump...
Target is MySQL 8.0.28. Dump was produced from MySQL 8.0.28
Scanning metadata - done
Checking for pre-existing objects...
Executing common preamble SQL
Executing DDL - done
Executing view DDL - done
Starting data load
1 thds loading \ 100% (2.93 MB / 2.93 MB), 1.16 MB/s, 0 / 1 tables done
Executing common postamble SQL
Recreating indexes - done
1 chunks (15.29K rows, 2.93 MB) for 1 tables in 1 schemas were loaded in 9 sec (avg throughput 1.16 MB/s)
0 warnings were reported during the load.

If the restore worked properly, the table sbtest1 should be restored. We now have recovered the table sbtest1 without the queries that have been executed afterwards (before the drop table).

 MySQL  albatroz.mysql.database.azure.com:3306 ssl  sysbench  SQL > show tables;
+--------------------+
| Tables_in_sysbench |
+--------------------+
| sbtest1            |
| sbtest10           |
| sbtest2            |
| sbtest3            |
| sbtest4            |
| sbtest5            |
| sbtest6            |
| sbtest7            |
| sbtest8            |
| sbtest9            |
+--------------------+
10 rows in set (0.1169 sec)

MySQL  albatroz.mysql.database.azure.com:3306 ssl  sysbench  SQL > select * from sbtest1 where pad like 'my%';
Empty set (0.1284 sec)

5. Execution of the binary logs

Before executing the binary logs, we need to define what we call the “start-position” and the “end-position“. In order to find these two numbers, we have to find the log position after the backup and to find the log position of the “drop table”. The first (start-position) can be found into the metadata of the dump ( .json file). For the second we have to find the exact position using mysqlbinlog as demonstrated below (drop position at 14329648)

mysqlbinlog --verify-binlog-checksum --host=albatroz.mysql.database.azure.com --port=3306 --user=grs -p -                                                   -read-from-remote-server --verbose --start-datetime="2022-08-19 18:40:40" --stop-datetime="2022-08-19 18:50:47" mysql-bin.00                                                   0006 | grep -C 15 "DROP TABLE"

# at 14329648
#220819 18:47:12 server id 3691359094  end_log_pos 14329725 CRC32 0xa81066cf    Anonymous_GTID  last_committed=6652     sequence_number=6653r                                  br_only=no      original_committed_timestamp=1660927632152016   immediate_commit_timestamp=1660927632152016     transaction_length=217
# original_commit_timestamp=1660927632152016 (2022-08-19 18:47:12.152016 CEST)
# immediate_commit_timestamp=1660927632152016 (2022-08-19 18:47:12.152016 CEST)
/*!80001 SET @@session.original_commit_timestamp=1660927632152016*//*!*/;
/*!80014 SET @@session.original_server_version=80028*//*!*/;
/*!80014 SET @@session.immediate_server_version=80028*//*!*/;
SET @@SESSION.GTID_NEXT= 'ANONYMOUS'/*!*/;
# at 14329725
#220819 18:47:12 server id 3691359094  end_log_pos 14329865 CRC32 0x9f7865b0    Query   thread_id=480   exec_time=0     error_code=0    Xid =                                   412469
use `sysbench`/*!*/;
SET TIMESTAMP=1660927632/*!*/;
DROP TABLE `sbtest1` /* generated by server */
/*!*/;

Now that we have the start and end position of the binary log, we can apply the events in binary log file to the server. For my part, I prefer to go through an intermediate step consisting in creating a file containing all the events. This way I can look at what is inside and then just run my file. This allows me, for example, to see if I have made an error. As shown below I’m for instance able to show my “insert” statement:

osboxes@osboxes:~$  mysqlbinlog --verify-binlog-checksum --host=albatroz.mysql.database.azure.com --port=3306 --user=grs -p --read-from-remote-server --start-datetime="2022-08-19 16:40:40" --stop-datetime="2022-08-19 16:46:47" mysql-bin.000006 >/tmp/restore.sql
Enter password:
osboxes@osboxes:~$ vi /tmp/restore.sql
...
# at 14329545
#220819 18:41:18 server id 3691359094  end_log_pos 14329617 CRC32 0xd124e6a0    Write_rows: table id 913 flags: STMT_END_F

BINLOG '
Lr3/YhN2qwXcRQAAAMmm2gAAAJEDAAAAAAEACHN5c2JlbmNoAAdzYnRlc3QxAAQDA/7+BO7g/vAA
AQEAAgP8/wDqexN/
Lr3/Yh52qwXcSAAAABGn2gAAAJEDAAAAAAEAAgAE/wD/yZo7AQAAAAEAMRhteSByb3cgYmVmb3Jl
IGRyb3AgdGFibGWg5iTR
'/*!*/;
### INSERT INTO `sysbench`.`sbtest1`
### SET
###   @1=999999999
###   @2=1
###   @3='1'
###   @4='my row before drop table'
# at 14329617
...

Finally we can execute our restore script on the database.

osboxes@osboxes:~$ mysql --host=albatroz.mysql.database.azure.com --port=3306 --user=grs -p </tmp/restore.sql
Enter password:

6. Let’s check that our last insert has been executed

As we can see, the last record we inserted in the table is now present.

 MySQL  albatroz.mysql.database.azure.com:3306 ssl  sysbench  SQL > select * from sbtest1 where pad like 'my%';
+-----------+---+---+--------------------------+
| id        | k | c | pad                      |
+-----------+---+---+--------------------------+
| 999999999 | 1 | 1 | my row before drop table |
+-----------+---+---+--------------------------+

Conclusion

Azure Backup Restore interface provides an easy and interesting solution to backup and restore a MySQL Server going through the replication of the server. In addition in the tests I did, the deployment of the new server was rather fast. However, my server did not contain gigabytes of information. The 35 days of backup window could be seen as too small depending on your needs.

Beside of this solution I strongly encourage Database Administrators to keep on backing up their MySQL Database using other tools such as the ones provided by MySQL Shell or any other backup solution in order to ensure that no transactions are lost when restoring data. Such solutions could offer more flexibility in the backup and restore process and longer retention.

L’article MySQL Server on Microsoft Azure 3rd part (backup and recovery) est apparu en premier sur dbi Blog.

Introduction

This second blog follows the first blog about deploying MySQL Server on Microsoft Azure. In the first blog, we saw how easy it is to deploy a MySQL server in minutes on the Azure cloud and we connected on it through the MySQL Shell client.

This second blog is more focused on the performance of MySQL in the Azure Cloud. Although I didn’t do any tuning of MySQL parameters, we will see the influence of MySQL Server localization on the latency as well as changes in parameters such as CPU, memory or IOPS on performances using the SysBench tool.

SysBench stress test

Of course there are many tools available to do stress tests. On my side, I decided to use SysBench simply because I know this wonderful free tool and because it is widely deployed. Rather than making a description of SysBench, I simply take a part of its description available on sourceforge that we can find below:

sysbench is a scriptable multi-threaded benchmark tool based on LuaJIT. It is most frequently used for database benchmarks, but can also be used to create arbitrarily complex workloads that do not involve a database server – https://sourceforge.net/projects/sysbench.mirror/, 14.08.2022

Firstly I installed SysBench on my local Ubuntu client (osboxes). After the installation of SysBench but before starting the stress test, we first have to prepare the tables containing the records where the queries will be performed. In the following tests I created 10 tables with 1’000’000 rows in sysbench database (previously created).

osboxes@osboxes:/usr/bin$ sysbench --db-driver=mysql table-size=1000000 --mysql-host=albatroz.mysql.database.azure.com --mysql-port=3306 --mysql-db=sysbench --mysql-user=grs --tables=10 --mysql-password=MyPassword --test=/usr/share/sysbench/oltp_read_write.lua prepare

Once the tables populated, we can run the tests. For the tests, I decided to use oltp_read_write.lua which has a default ratio of read:write 95%:5%.

sysbench --db-driver=mysql  --num-threads=8 --mysql-user=grs --mysql-password=MyPassword --mysql-db=sysbench --events=0 --time=100  --test=/usr/share/sysbench/oltp_read_write.lua --mysql-host=albatroz.mysql.database.azure.com --mysql-port=3306 --tables=10 --db-ps-mode=disable --table-size=1000000 --report-interval=10 run

I varied the number of threads (num-threads parameter) from 8 to 256, limited the time to 100 seconds and changed two kinds of parameters through the Azure interface during my tests:

• The Server location (East US, vs West Switzerland)
• The Compute and Storage (Compute Tier and IOPS)

It’s important to note that my Client is also located in Western Switzerland to understand the impact of location. You can find below the four configurations I did:

1. Server located in US, with minimal compute and storage meaning 1vCore 2Gib Memory and 360 IOPS
2. Server located in US, with general purpose configuration meaning 2vCores, 8Gib Memory and 3200 IOPS
3. Server located in West Switzerland, with minimal compute and storage meaning 1vCore 2Gib Memory and 360 IOPS
4. Server located in West Switzerland, with general purpose configuration meaning 2vCores, 8Gib Memory and 3200 IOPS

Server details

As it’s not possible to change the location of a server afterwards, I created two MySQL servers in same version (version 8.0.28). One located in East US and the other located in West Switzerland. Note that we have two locations in Switzerland (North Switzerland and West Switzerland). As explained above, in order to make comparisons, I provided the same set of resources for tests 1 and 3 and test 2 and 4. Also as specified before, my client is located in Switzerland.

The basic configuration looks like the one below:

It is worth mentioning that to upgrade the server’s capacity (from Burstable to General Purpose with more IOPS), it only takes about ten minutes maximum. Meaning that when we end with the tests and we want to go to production, we only need few minutes to upgrade the server properties. We can find the description of the upgraded server below as well as the monthly price of this upgraded server.

Let’s run the tests

I executed the same tests with only a variation in the number of thread (from 8 to 256) on each four configurations. We can find an extract of the execution as well as the output below

osboxes@osboxes:/usr/bin$ sysbench --db-driver=mysql  --num-threads=8 --mysql-user=grs --mysql-password=MyPassword --mysql-db=sysbench --events=0 --time=100  --test=/usr/share/sysbench/oltp_read_write.lua --mysql-host=albatroz.mysql.database.azure.com --mysql-port=3306 --tables=10 --db-ps-mode=disable --table-size=1000000 --report-interval=10 run
WARNING: the --test option is deprecated. You can pass a script name or path on the command line without any options.
WARNING: --num-threads is deprecated, use --threads instead
sysbench 1.0.20 (using system LuaJIT 2.1.0-beta3)

Running the test with following options:
Number of threads: 8
Report intermediate results every 10 second(s)
Initializing random number generator from current time


Initializing worker threads...

Threads started!

[ 10s ] thds: 8 tps: 3.20 qps: 77.58 (r/w/o: 55.98/4.40/17.20) lat (ms,95%): 2082.91 err/s: 0.00 reconn/s: 0.00
[ 20s ] thds: 8 tps: 4.00 qps: 72.91 (r/w/o: 50.51/5.00/17.40) lat (ms,95%): 2585.31 err/s: 0.00 reconn/s: 0.00
[ 30s ] thds: 8 tps: 4.00 qps: 77.80 (r/w/o: 53.80/5.30/18.70) lat (ms,95%): 2120.76 err/s: 0.00 reconn/s: 0.00
[ 40s ] thds: 8 tps: 3.20 qps: 72.80 (r/w/o: 52.50/4.50/15.80) lat (ms,95%): 2632.28 err/s: 0.00 reconn/s: 0.00
[ 50s ] thds: 8 tps: 4.00 qps: 78.40 (r/w/o: 55.40/5.10/17.90) lat (ms,95%): 2082.91 err/s: 0.00 reconn/s: 0.00
[ 60s ] thds: 8 tps: 4.00 qps: 73.60 (r/w/o: 49.70/5.40/18.50) lat (ms,95%): 2632.28 err/s: 0.00 reconn/s: 0.00
[ 70s ] thds: 8 tps: 4.00 qps: 77.60 (r/w/o: 53.60/5.00/19.00) lat (ms,95%): 2082.91 err/s: 0.00 reconn/s: 0.00
[ 80s ] thds: 8 tps: 3.20 qps: 73.60 (r/w/o: 54.10/4.70/14.80) lat (ms,95%): 2405.65 err/s: 0.00 reconn/s: 0.00
[ 90s ] thds: 8 tps: 4.00 qps: 77.60 (r/w/o: 53.90/4.80/18.90) lat (ms,95%): 2082.91 err/s: 0.00 reconn/s: 0.00
[ 100s ] thds: 8 tps: 4.00 qps: 73.60 (r/w/o: 49.60/4.80/19.20) lat (ms,95%): 2493.86 err/s: 0.00 reconn/s: 0.00
SQL statistics:
    queries performed:
        read:                            5376
        write:                           501
        other:                           1803
        total:                           7680
    transactions:                        384    (3.78 per sec.)
    queries:                             7680   (75.60 per sec.)
    ignored errors:                      0      (0.00 per sec.)
    reconnects:                          0      (0.00 per sec.)

General statistics:
    total time:                          101.5914s
    total number of events:              384

Latency (ms):
         min:                                 2000.71
         avg:                                 2115.55
         max:                                 2644.11
         95th percentile:                     2585.31
         sum:                               812371.47

Threads fairness:
    events (avg/stddev):           48.0000/0.00
    execution time (avg/stddev):   101.5464/0.07

We can see in this output, that I asked to sysbench to provide me with a feedback every 10 seconds. We can also observe that the ratio between the queries and transaction is respected (95%:5%), finally it can noted that the latency is important in our case (Server in US and client in Switzerland). The average latency in the above example is 2’115.55ms. If we are interested in the latency distribution we can use the option –histogram.

Tests summary

We can find below the summary of the tests I did with the described configurations.

The first thing we can see after having a look on the above table is that whereas we faced a “too many connections” errors with minimal configurations, the tests worked properly with the General purpose configuration.

The second thing that we can observe is that the tests performed better with the server located in West Switzerland. One of the reasons is most probably linked with the latency. Whereas we had an average of 2’100ms of latency for server located in East US, we have an average of 350ms of latency for server located in West Switzerland.

Finally in our configuration, the increase of compute and storage doesn’t show any improvement in terms of transaction per second or queries per second.

Latency between the application and the MySQL Server

The latency is an important parameter to take into consideration. As explained on Microsoft webpage named “Best practices for optimal performance of your Azure Database for MySQL server” to improve the performance of an application, we have to take into consideration the proximity between the MySQL Server and the application.

To improve the performance and reliability of an application in a cost optimized deployment, it’s highly recommended that the web application service and the Azure Database for MySQL resource reside in the same region and availability zone. – Microsoft, 16.08.2022

This page also gives some tips in order to optimize your MySQL performances on Azure. To check the latency, you do not need to use SysBench, a simple query such as the one executed in the below examples presents the latency difference between the execution on a localhost, an Azure server located in Switzerland and an Azure server located in US.

Latency on a Localhost

mysql> select 1;
+---+
| 1 |
+---+
| 1 |
+---+
1 row in set (0.00 sec)

Latency on an Azure Server located in West Switzerland

 MySQL  albatrozswitzerland.mysql.database.azure.com:3306 ssl SQL > select 1;
+---+
| 1 |
+---+
| 1 |
+---+
1 row in set (0.0436 sec)

Latency on an Azure Server located in East US

 MySQL  albatroz.mysql.database.azure.com:3306 ssl SQL > select 1;
+---+
| 1 |
+---+
| 1 |
+---+
1 row in set (0.1690 sec)

Considering only the latency, one could think that the application must be hosted on an Azure server in Switzerland (at least in my case). Unfortunately we will discover that not all services of the Microsoft Azure Cloud are available in Switzerland. For instance if we want to deploy an Ubuntu Server, we will see that only the following location are available:

Performance monitoring

Microsoft Azure provides us with an interface showing the following default graphs (cf screenshot below:

• CPU and Memory
• IO Percent
• DB Connections
• Queries

But you can add other information regarding storage, Host network in/out, replication lag, aborted connection, aso…

Conclusion

This short blog regarding the performance of MySQL in the Azure Cloud provides the following conclusions:

The ability to upgrade a MySQL server by adding CPU, disk and memory capacity in a few minutes is really interesting. For instance when you want to switch your test project to production, you simply need to change the servers properties and 10 minutes later, your server is adapted. Additionally, we can always reduce the CPU & Memory capacity as well as the IOPS, only the storage cannot be reduced.

At first glance, it also seems that it is very important to use a MySQL server close to the client for performance reasons (in order to avoid latency issues). Unfortunately depending on the application and business requirements it could be difficult to have the MySQL Server and the Application Server in the same country.

Finally adding CPU, memory and increasing the number of IOPS did not show any improvement in the number of transactions or queries per second in this specific configuration (of course it’s strongly depending on the use case). However, this does not mean that by taking the time to configure the MySQL server correctly, one could not benefit from these additional capacities. Regarding the tuning of MySQL variables, it’s interesting to see that some variables are updated automatically such as the innodb_buffer_pool_size when we change server resources.

L’article MySQL Server on Microsoft Azure 2nd part (Performance tests) est apparu en premier sur dbi Blog.

Introduction

Did you know that you can run MySQL on Microsoft Azure for free for 30 days with a $200 credit? In this first blog I’ll show how to create a MySQL Server and provide few information related to this service. In futures blogs I’ll present insights regarding MySQL performances and backup/recovery.

Let’s start by registering ourselves

The registration process lasts approximately 5 to 10 minutes. We simply have to enter our contacts details as well as our credit card details. Don’t worry, no fees are automatically charged. Once your 200$ credit limit used, Microsoft will ask us if we want to keep on with the payment of the additional fees.  

In order to register ourselves on Microsoft Azure, we can simply go on the following URL: https://azure.microsoft.com/en-us/services/mysql/ and click on “Try Azure Database for MySQL free” and “Start free”. Then we enter our contact details such as country, name, surname, phone number, e-mail address, address, aso… Once finished, we will see a screen looking like the one below. We simply have to select the time zone and if we want to join a Q&A session.

Our first MySQL Server on Azure

Once the registration process ended, it’s time to create our first MySQL Server on Azure. Log in into Microsoft Azure and enter “mysql” in the research field as shown below and select “Azure Database for MySQL servers”.  

Once “Azure Database for MySQL servers” selected, we simply have to click on “Create Azure Database for MySQL Server” on the next screen. We now have to select the deployment option. As shown below, we have the choice between the two following options:

• Flexible Server (Recommended)
• Single Server

In the context of this blog, I chose the first option (Flexible server).

Before deploying our first MySQL Server we have to follow a straightforward process which requires 4 steps described below. As we will see, depending on the workload type retained (which will affect the CPU, memory and possible IOPS), the storage and IOPS, the monthly fees will vary between USD 18.98/month and USD >10’000/month.

What is really nice is to be able to see in real time the influence that each parameter has on the price estimate on the right hand side as we can see in the screenshot below.

1. Firstly, we have to enter basic information regarding our MySQL Server such as :
   • Subscription details,
   • Server name
   • Region where you want to deploy your server
   • MySQL Version (version 5.x or 8.x)
   • Workload type (Small or Medium Size database, Tier 1 Business Critical Workloads, for development of hobby projects)
   • Compute+Storage: Compute options – (Burstable 1-20 vCores, General purpose 2-64 vCores or Business Critical 2-96 vCores). Storage options – from 20GB up to 16384GB (no scale down possibility) and from 360 to 48000IOPS
   • Availability Zone (optional)
   • Enable High-Availability (optional)
   • Administrator account
     
     
2. Secondly, we have to fill in networking information regarding our MySQL Server such as:
   • Network Connectivity (connect to your server through a public IP address or a Private Access)
   • Firewall rules (you can automatically select your current IP address or allow any remote connection)
     
     
3. Thirdly, we can enter tags. Tags are name/value pairs that enable you to categorize and view consolidated billing by applying the same tag to multiple resources and resource groups.
   
   
4. We can finally review and create our server. Please note that we can also download a template for automation purpose.

In the compute option, you can select few options as presented below:

Once all the the fields completed we can create our server. It’s interesting to see that we also have the possibility to generate a template for automation purpose.

On the summary page, you can see that you have access to the Terms of Use and Privacy Policy. In the Terms of Use we can find a link to the SLA Conditions where we can find the guaranteed service time as well as the credit in case of non compliance with the SLA conditions.

The deployment and start-up of the server takes approximately 6 minutes.

We have a server, let’s use it now!

In order to use our brand new MySQL Server, I decided to use MySQL Shell. MySQL Shell is a MySQL Client that provides scripting capabilities in JavaScript and Python. You can download MySQL Shell on this link and find MySQL Documentation on this link.

When we run MySQL Shell on Windows, a cmd window opens with a command prompt as shown below. To connect to our MySQL Server, we can use the “connect” keyword. and then switch in sql mode with the “\sql” command.

MySQL  JS > \connect --mysql albatroz.mysql.database.azure.com
Creating a Classic session to 'grs@albatroz.mysql.database.azure.com'
Please provide the password for 'grs@albatroz.mysql.database.azure.com': *************
Save password for 'grs@albatroz.mysql.database.azure.com'? [Y]es/[N]o/Ne[v]er (default No): yes
Fetching schema names for autocompletion... Press ^C to stop.
Your MySQL connection id is 20
Server version: 8.0.28 Source distribution
No default schema selected; type \use <schema> to set one.
 MySQL  albatroz.mysql.database.azure.com:3306 ssl  JS > \sql
Switching to SQL mode... Commands end with ;

Using show variables like ‘%version%’; command provides us few information regarding the MySQL version as well as the Operating System in use. Version 8.0.28 is not the latest version but it is not old either since it dates from December 2021 and at the time of writing this blog, the latest version is 8.0.30.

MySQL  albatroz.mysql.database.azure.com:3306 ssl  SQL > show variables like '%version%';
+--------------------------+---------------------+
| Variable_name            | Value               |
+--------------------------+---------------------+
| admin_tls_version        | TLSv1.2             |
| immediate_server_version | 999999              |
| innodb_version           | 8.0.28              |
| original_server_version  | 999999              |
| protocol_version         | 10                  |
| replica_type_conversions |                     |
| slave_type_conversions   |                     |
| tls_version              | TLSv1.2             |
| version                  | 8.0.28              |
| version_comment          | Source distribution |
| version_compile_machine  | x86_64              |
| version_compile_os       | Linux               |
| version_compile_zlib     | 1.2.11              |
+--------------------------+---------------------+

After few tests, we can see some performance graphs (CPU and Memory, IO Percent, DB Connections and Queries) through the Azure interface.

Conclusion

This first blog does not pretend to go into detail about the possibilities offered by Azure for the deployment of a MySQL server. It simply shows that deploying a MySQL server on the Azure Cloud is really simple. In addition, the $200 offered by Microsoft on the first 30 days of use allows you to have a first overview of the possibilities but also the costs related to a deployment in the Azure Cloud. The interface allows anyone to easily deploy a MySQL database in minutes.

The next blog will discuss MySQL performance on the Azure Cloud through stress tests performed with SysBench.

L’article MySQL Server on Microsoft Azure 1st part (deployment) est apparu en premier sur dbi Blog.

In general, I would like to raise two points before going into the technical details:
1. Is it a good idea to move databases into containers? Here I would use a typical IT answer: “it depends”. I can suggest you to think about your needs and constraints, if you have small images to deploy, about storage and persistence, performances, …
2. There are various solutions for installing, orchestrating and administering MySQL in K8s: MySQL single instance vs MySQL InnoDB Cluster, using MySQL Operator for Kubernetes or Helm Charts, on-premise but also through Oracle Container Engine for Kubernetes on OCI, … I recommend you to think about which are (again) your needs and skills, if you are already working on Cloud technologies, whether you have already set up DevOps processes and which ones, …

Here I will show you how to install a MySQL InnoDB Cluster in OKE using a MySQL Operator.

First thing is to have an account on Oracle OCI and have deployed an Oracle Container Engine for Kubernetes in your compartment. You can do it in an easy was using the Quick Create option under “Developer Services > Containers & Artifacts > Kubernetes Clusters (OKE)”:

In this way all the resources you need (VCN, Internet and NAT gateways, a K8s cluster with workers nodes and node pool) are there in one click:

elisa@cloudshell:~ (eu-zurich-1)$ kubectl cluster-info
Kubernetes control plane is running at https://xxx.xx.xxx.xxx:6443
CoreDNS is running at https://xxx.xx.xxx.xxx:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

elisa@cloudshell:~ (eu-zurich-1)$ kubectl get nodes -o wide
NAME         STATUS   ROLES   AGE    VERSION   INTERNAL-IP   EXTERNAL-IP       OS-IMAGE                  KERNEL-VERSION                      CONTAINER-RUNTIME
10.0.10.36   Ready    node    6m7s   v1.22.5   10.0.10.36    yyy.yyy.yyy.yyy   Oracle Linux Server 7.9   5.4.17-2136.304.4.1.el7uek.x86_64   cri-o://1.22.3-1.ci.el7
10.0.10.37   Ready    node    6m1s   v1.22.5   10.0.10.37    kkk.kkk.kkk.kk    Oracle Linux Server 7.9   5.4.17-2136.304.4.1.el7uek.x86_64   cri-o://1.22.3-1.ci.el7
10.0.10.42   Ready    node    6m     v1.22.5   10.0.10.42    jjj.jj.jjj.jj     Oracle Linux Server 7.9   5.4.17-2136.304.4.1.el7uek.x86_64   cri-o://1.22.3-1.ci.el7

As a second step, you can install the MySQL Operator for Kubernetes using kubectl:

elisa@cloudshell:~ (eu-zurich-1)$ kubectl apply -f https://raw.githubusercontent.com/mysql/mysql-operator/trunk/deploy/deploy-crds.yaml
customresourcedefinition.apiextensions.k8s.io/innodbclusters.mysql.oracle.com created
customresourcedefinition.apiextensions.k8s.io/mysqlbackups.mysql.oracle.com created
customresourcedefinition.apiextensions.k8s.io/clusterkopfpeerings.zalando.org created
customresourcedefinition.apiextensions.k8s.io/kopfpeerings.zalando.org created
elisa@cloudshell:~ (eu-zurich-1)$ kubectl apply -f https://raw.githubusercontent.com/mysql/mysql-operator/trunk/deploy/deploy-operator.yaml
serviceaccount/mysql-sidecar-sa created
clusterrole.rbac.authorization.k8s.io/mysql-operator created
clusterrole.rbac.authorization.k8s.io/mysql-sidecar created
clusterrolebinding.rbac.authorization.k8s.io/mysql-operator-rolebinding created
clusterkopfpeering.zalando.org/mysql-operator created
namespace/mysql-operator created
serviceaccount/mysql-operator-sa created
deployment.apps/mysql-operator created

You can check the health of the MySQL Operator:

elisa@cloudshell:~ (eu-zurich-1)$ kubectl get deployment -n mysql-operator mysql-operator
NAME             READY   UP-TO-DATE   AVAILABLE   AGE
mysql-operator   1/1     1            1           24s
elisa@cloudshell:~ (eu-zurich-1)$ kubectl get pods --show-labels -n mysql-operator
NAME                              READY   STATUS    RESTARTS   AGE    LABELS
mysql-operator-869d4b4b8d-slr4t   1/1     Running   0          113s   name=mysql-operator,pod-template-hash=869d4b4b8d

To isolate resources, you can create a dedicated namespace for the MySQL InnoDB Cluster:

elisa@cloudshell:~ (eu-zurich-1)$ kubectl create namespace mysql-cluster
namespace/mysql-cluster created

You should also create a Secret using kubectl to store MySQL user credentials that will be created and then required by pods to access to the MySQL server:

elisa@cloudshell:~ (eu-zurich-1)$ kubectl create secret generic elisapwd --from-literal=rootUser=root --from-literal=rootHost=% --from-literal=rootPassword="pwd" -n mysql-cluster
secret/elisapwd created

You can check that the Secret was corrected created:

elisa@cloudshell:~ (eu-zurich-1)$ kubectl get secrets -n mysql-cluster
NAME                  TYPE                                  DATA   AGE
default-token-t2c47   kubernetes.io/service-account-token   3      2m
elisapwd              Opaque                                3      34s
elisa@cloudshell:~ (eu-zurich-1)$ kubectl describe secret/elisapwd -n mysql-cluster
Name:         elisapwd
Namespace:    mysql-cluster
Labels:       
Annotations:  

Type:  Opaque

Data
====
rootHost:      1 bytes
rootPassword:  7 bytes
rootUser:      4 bytes

Now you have to write a .yaml configuration file to define how the MySQL InnoDB Cluster should be created. Here is a simple example:

elisa@cloudshell:~ (eu-zurich-1)$ vi InnoDBCluster_config.yaml
apiVersion: mysql.oracle.com/v2alpha1
kind: InnoDBCluster
metadata:
  name: elisacluster
  namespace: mysql-cluster 
spec:
  secretName: elisapwd
  instances: 3
  router:
    instances: 1

At this point you can run a MySQL InnoDB Cluster applying the configuration that you just created:

elisa@cloudshell:~ (eu-zurich-1)$ kubectl apply -f InnoDBCluster_config.yaml
innodbcluster.mysql.oracle.com/elisacluster created

You can finally check if the MySQL InnoDB Cluster has been successfully created:

elisa@cloudshell:~ (eu-zurich-1)$ kubectl get innodbcluster --watch --namespace mysql-cluster
NAME           STATUS    ONLINE   INSTANCES   ROUTERS   AGE
elisacluster   PENDING   0        3           1         12s
elisacluster   PENDING   0        3           1         103s
elisacluster   INITIALIZING   0        3           1         103s
elisacluster   INITIALIZING   0        3           1         103s
elisacluster   INITIALIZING   0        3           1         103s
elisacluster   INITIALIZING   0        3           1         104s
elisacluster   INITIALIZING   0        3           1         106s
elisacluster   ONLINE         1        3           1         107s
elisa@cloudshell:~ (eu-zurich-1)$ kubectl get all -n mysql-cluster
NAME                                       READY   STATUS    RESTARTS   AGE
pod/elisacluster-0                         2/2     Running   0          4h44m
pod/elisacluster-1                         2/2     Running   0          4h42m
pod/elisacluster-2                         2/2     Running   0          4h41m
pod/elisacluster-router-7686457f5f-hwfcv   1/1     Running   0          4h42m

NAME                             TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)                               AGE
service/elisacluster             ClusterIP   10.96.9.203           6446/TCP,6448/TCP,6447/TCP,6449/TCP   4h44m
service/elisacluster-instances   ClusterIP   None                  3306/TCP,33060/TCP,33061/TCP          4h44m

NAME                                  READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/elisacluster-router   1/1     1            1           4h44m

NAME                                             DESIRED   CURRENT   READY   AGE
replicaset.apps/elisacluster-router-7686457f5f   1         1         1       4h44m

NAME                            READY   AGE
statefulset.apps/elisacluster   3/3     4h44m

You can use port forwarding in the following way:

elisa@cloudshell:~ (eu-zurich-1)$ kubectl port-forward service/elisacluster mysql --namespace=mysql-cluster
Forwarding from 127.0.0.1:6446 -> 6446

to access your MySQL InnoDB Cluster on a second terminal in order to check its health:

elisa@cloudshell:~ (eu-zurich-1)$ mysqlsh -h127.0.0.1 -P6446 -uroot -p
Please provide the password for 'root@127.0.0.1:6446': *******
Save password for 'root@127.0.0.1:6446'? [Y]es/[N]o/Ne[v]er (default No): N
MySQL Shell 8.0.28-commercial

Copyright (c) 2016, 2022, Oracle and/or its affiliates.
Oracle is a registered trademark of Oracle Corporation and/or its affiliates.
Other names may be trademarks of their respective owners.

Type 'help' or '?' for help; 'quit' to exit.
Creating a session to 'root@127.0.0.1:6446'
Fetching schema names for autocompletion... Press ^C to stop.
Your MySQL connection id is 36651
Server version: 8.0.28 MySQL Community Server - GPL
No default schema selected; type use  to set one.
 MySQL  127.0.0.1:6446 ssl  JS >  MySQL  127.0.0.1:6446 ssl  JS > dba.getCluster().status();
{
    "clusterName": "elisacluster", 
    "defaultReplicaSet": {
        "name": "default", 
        "primary": "elisacluster-0.elisacluster-instances.mysql-cluster.svc.cluster.local:3306", 
        "ssl": "REQUIRED", 
        "status": "OK", 
        "statusText": "Cluster is ONLINE and can tolerate up to ONE failure.", 
        "topology": {
            "elisacluster-0.elisacluster-instances.mysql-cluster.svc.cluster.local:3306": {
                "address": "elisacluster-0.elisacluster-instances.mysql-cluster.svc.cluster.local:3306", 
                "memberRole": "PRIMARY", 
                "memberState": "(MISSING)", 
                "mode": "n/a", 
                "readReplicas": {}, 
                "role": "HA", 
                "shellConnectError": "MySQL Error 2005: Could not open connection to 'elisacluster-0.elisacluster-instances.mysql-cluster.svc.cluster.local:3306': Unknown MySQL server host 'elisacluster-0.elisacluster-instances.mysql-cluster.svc.cluster.local' (-2)", 
                "status": "ONLINE", 
                "version": "8.0.28"
            }, 
            "elisacluster-1.elisacluster-instances.mysql-cluster.svc.cluster.local:3306": {
                "address": "elisacluster-1.elisacluster-instances.mysql-cluster.svc.cluster.local:3306", 
                "memberRole": "SECONDARY", 
                "memberState": "(MISSING)", 
                "mode": "n/a", 
                "readReplicas": {}, 
                "role": "HA", 
                "shellConnectError": "MySQL Error 2005: Could not open connection to 'elisacluster-1.elisacluster-instances.mysql-cluster.svc.cluster.local:3306': Unknown MySQL server host 'elisacluster-1.elisacluster-instances.mysql-cluster.svc.cluster.local' (-2)", 
                "status": "ONLINE", 
                "version": "8.0.28"
            }, 
            "elisacluster-2.elisacluster-instances.mysql-cluster.svc.cluster.local:3306": {
                "address": "elisacluster-2.elisacluster-instances.mysql-cluster.svc.cluster.local:3306", 
                "memberRole": "SECONDARY", 
                "memberState": "(MISSING)", 
                "mode": "n/a", 
                "readReplicas": {}, 
                "role": "HA", 
                "shellConnectError": "MySQL Error 2005: Could not open connection to 'elisacluster-2.elisacluster-instances.mysql-cluster.svc.cluster.local:3306': Unknown MySQL server host 'elisacluster-2.elisacluster-instances.mysql-cluster.svc.cluster.local' (-2)", 
                "status": "ONLINE", 
                "version": "8.0.28"
            }
        }, 
        "topologyMode": "Single-Primary"
    }, 
    "groupInformationSourceMember": "elisacluster-0.elisacluster-instances.mysql-cluster.svc.cluster.local:3306"
}

 MySQL  127.0.0.1:6446 ssl  JS > sql
Switching to SQL mode... Commands end with ;
 MySQL  127.0.0.1:6446 ssl  SQL > select @@hostname;
+----------------+
| @@hostname     |
+----------------+
| elisacluster-0 |
+----------------+
1 row in set (0.0018 sec)
 MySQL  127.0.0.1:6446 ssl  SQL > SELECT * FROM performance_schema.replication_group_members;
+---------------------------+--------------------------------------+-----------------------------------------------------------------------+-------------+--------------+-------------+----------------+----------------------------+
| CHANNEL_NAME              | MEMBER_ID                            | MEMBER_HOST                                                           | MEMBER_PORT | MEMBER_STATE | MEMBER_ROLE | MEMBER_VERSION | MEMBER_COMMUNICATION_STACK |
+---------------------------+--------------------------------------+-----------------------------------------------------------------------+-------------+--------------+-------------+----------------+----------------------------+
| group_replication_applier | 717dbe17-ba71-11ec-8a91-3665daa9c822 | elisacluster-0.elisacluster-instances.mysql-cluster.svc.cluster.local |        3306 | ONLINE       | PRIMARY     | 8.0.28         | XCom                       |
| group_replication_applier | b02c3c9a-ba71-11ec-8b65-5a93db09dda5 | elisacluster-1.elisacluster-instances.mysql-cluster.svc.cluster.local |        3306 | ONLINE       | SECONDARY   | 8.0.28         | XCom                       |
| group_replication_applier | eb06aadd-ba71-11ec-8aac-aa31e5d7e08b | elisacluster-2.elisacluster-instances.mysql-cluster.svc.cluster.local |        3306 | ONLINE       | SECONDARY   | 8.0.28         | XCom                       |
+---------------------------+--------------------------------------+-----------------------------------------------------------------------+-------------+--------------+-------------+----------------+----------------------------+
3 rows in set (0.0036 sec)

Easy, right?
Yes, but databases containers is still a tricky subject. As we said above, many topics need to be addressed: deployment type, performances, backups, storage and persistence, … So stay tuned, more blog posts about MySQL on K8s will come soon…

By Elisa Usai

L’article Installing MySQL InnoDB Cluster in OKE using a MySQL Operator est apparu en premier sur dbi Blog.

It’s Open Source!

Even if money is not everything…

PMM is completely Open Source, and this is not surprising coming from Percona. But as we know, the advantage of Open Source is not only the fact of being free…

Let’s contribute

One of the keys of Open Source is contribution. When I started using PMM and sometimes I cannot find all the information I needed and more details in the official documentation, or when I discover a bug or have a question, or even if I would like to have a new feature implemented, a whole community is there to support and share knowledge in a very impressive short time:
– A JIRA issue tracker to submit a bug report.
– A very useful blog with plenty of posts about new features, technical step-by-step deployment information, tips and much more. And obviously the possibility to write to the author.
– A Percona Community Forum to exchange with Percona experts.

Evolution

Open Source means also quick evolution of the product. PMM is getting better every month, with continuous improvements, new features, corrections of bugs and so on.

One tool to monitor several technologies

If we speak about monitoring solutions, what makes the difference nowadays? Except rare cases, each of us works with a mix of different database solutions: Open Source vs proprietary, On-Premises vs Cloud. And when we want to identify performance issues on our application, we also need system benchmarks, replication metrics, etc… PMM is a platform that we can use to centralize our monitoring data, because:
– Exporters to collect metrics for MySQL, MariaDB, PostgreSQL and MongoDB instances and their replication, the ProxySQL and the HAProxy, instances hosted on Amazon RDS or on Google Cloud Platform are embedded in the PMM tool as well as pre-configured dashboards, so the monitoring can be easily put in place.
– Even proprietary technologies such as Oracle and SQL Server can be integrated to PMM through external exporters. In this case we have the choice to develop our own exporter and design our own dashboards or use what is already been developed and designed by other contributors. Here you can find the links to some Third-party exporters.

Less maintenance tasks

The PMM platform is based on a client-server model. The 2 main components on the server side are the following ones:
– VictoriaMetrics, a time series database (replacing Prometheus since PMM 2.12.0 version) that aggregates metrics collected by exporters
– Grafana allowing to visualize the aggregated data in a web interface.
But actually, we don’t have to care about that! We don’t need to install Prometheus or VictoriaMetrics components, install Grafana and then configure it to let them talk together. We can just see the PMM platform as a black box: we will have only one tool to install and to maintain.

Sexy dashboards

Integrated dashboards are designed with accuracy and provide a detailed and temporal analysis of our data.
Here my favorite ones:
– Node Summary
It’s the system dashboard. We can visualize here metrics such as CPU, memory, disk occupation and performances, processes, network traffic and get some information about our system architecture:

We have also the possibility to compare several nodes:

– MySQL Summary
It’s one of the MySQL dashboards which displays general metrics about our instance (uptime, version, InnoDB Buffer Pool size, connections, threads, table locks, traffic, and much more):

And we still have the option to compare different instances.
– MySQL Replication
It’s the dashboard which offers us an overview on the MySQL Master-Slave Replication:

– Query Analytics
QAN is a special dashboard that allows us to analyze MySQL and PostgreSQL databases queries over time:

This can be very useful to identify performance problems.

Coming soon

Currently the last PMM stable release is 2.25.0. As we told just above, the tool is evolving a lot, and nice features are already available in technical preview:

You can already play with that even if for the moment it’s not suggested to use them for production environments. I tested for example the Integrated Alerting and I cannot wait to have it available as a stable feature (we don’t need then anymore to configure an External Prometheus Alertmanager to be alerted if something goes wrong on our systems).

I hope you have enjoyed this introduction to PMM.
Now it’s up to you: don’t hesitate to test it! For my part, I will surely write again about PMM, so stay tuned!

L’article What I really like about Percona PMM est apparu en premier sur dbi Blog.

.
In the NoSQL times, it was common to hear thinks like “SQL is bad”, “joins are bad”, “foreign keys are bad”. Just because people didn’t know how to use them, or they were running on a database system with a poor implementation of it. MySQL was very popular because easy to install, but lacking on many optimization features that you find in other open source or commercial databases. Sometimes, I even wonder if this NoSQL thing was not just a NoMySQL at its roots. When people encountered the limitations of MySQL and thought that it was SQL that was limited.

The following twitter thread, and linked articles, mention how DML on a child table can be blocked by DML on the parent. This is not a problem in some occasions (when this parent-child relationship is a composition where you work on the whole within the same transaction) but can be a problem when the parent is shared my many unrelated transactions.

Thoughts on foreign keys? The comment from @ShlomiNoach goes over some interesting points on why foreign keys should not be used: https://t.co/b6XG1oWRmb

— Fatih Arslan (@fatih) March 16, 2021

Sample Data

Surprised because I’ve not seen that even in the earliest versions of Oracle, I had to test it. Especially because MySQL and the InnoDB engines have evolved a lot since version 5. I’ll use Gerald Venzl sample data on countries and cities: https://github.com/gvenzl/sample-data/tree/master/countries-cities-currencies because he provides a SQL script that works on all databases without any changes.

MySQL 8.0.23

{
echo "create database if not exists countries;"
echo "use countries;"
curl -s https://raw.githubusercontent.com/gvenzl/sample-data/master/countries-cities-currencies/install.sql
} | mysql

This creates the tables and data with countries, and cities. There is a foreign key from cities that references countries.

Session 1: update the parent row

use countries;
begin;
select * from countries where country_code='CH';
update countries set population=population+1 where country_code='CH';

I’ve updated some info in countries, leaving the transaction opened.

mysql> begin;
Query OK, 0 rows affected (0.00 sec)

mysql> select * from countries where country_code='CH';
+------------+--------------+-------------+---------------------+------------+------------+----------+-----------+---------------+-----------+
| country_id | country_code | name        | official_name       | population | area_sq_km | latitude | longitude | timezone      | region_id |
+------------+--------------+-------------+---------------------+------------+------------+----------+-----------+---------------+-----------+
| CHE        | CH           | Switzerland | Swiss Confederation |    8293000 |   41277.00 | 47.00016 |   8.01427 | Europe/Zurich | EU        |
+------------+--------------+-------------+---------------------+------------+------------+----------+-----------+---------------+-----------+
1 row in set (0.00 sec)

mysql> update countries set population=population+1 where country_code='CH';
Query OK, 1 row affected (0.00 sec)
Rows matched: 1  Changed: 1  Warnings: 0

mysql> select * from countries where country_code='CH';
+------------+--------------+-------------+---------------------+------------+------------+----------+-----------+---------------+-----------+
| country_id | country_code | name        | official_name       | population | area_sq_km | latitude | longitude | timezone      | region_id |
+------------+--------------+-------------+---------------------+------------+------------+----------+-----------+---------------+-----------+
| CHE        | CH           | Switzerland | Swiss Confederation |    8293001 |   41277.00 | 47.00016 |   8.01427 | Europe/Zurich | EU        |
+------------+--------------+-------------+---------------------+------------+------------+----------+-----------+---------------+-----------+
1 row in set (0.00 sec)

With this transaction still on-going, I’ll insert, from another session, a new child row for this parent value: another country in Switzerland

Session 2: insert a child

use countries;
begin;
select * from cities where country_id='CHE';
insert into cities values ('CHE1170', 'Aubonne', null, 2750, 'N', 46.49514, 6.39155, null, 'CHE');

This blocks for a while… innodb-lock-wait-timeout defaults to 50 seconds.

Here is the output:

mysql> begin;
Query OK, 0 rows affected (0.00 sec)

mysql> select * from cities where country_id='CHE';
+---------+------+---------------+------------+------------+----------+-----------+----------+------------+
| city_id | name | official_name | population | is_capital | latitude | longitude | timezone | country_id |
+---------+------+---------------+------------+------------+----------+-----------+----------+------------+
| CHE0001 | Bern | NULL          |     422000 | Y          | 46.94809 |   7.44744 | NULL     | CHE        |
+---------+------+---------------+------------+------------+----------+-----------+----------+------------+
1 row in set (0.00 sec)

mysql> insert into cities values ('CHE1170', 'Aubonne', null, 2750, 'N', 46.49514, 6.39155, null, 'CHE');

ERROR 1205 (HY000): Lock wait timeout exceeded; try restarting transaction
mysql>

This insert tries to lock the parent row in share mode, and because it is currently being updated by session 1, the lock cannot be acquired. This is a very simple case of contention that can happen in many cases. Still there in MySQL 8

PostgreSQL

curl -s https://raw.githubusercontent.com/gvenzl/sample-data/master/countries-cities-currencies/install.sql | psql

This creates the tables and data with countries, and cities. There is a foreign key from cities that references countries.

Session 1: update the parent row

begin transaction;
select * from countries where country_code='CH';
update countries set population=population+1 where country_code='CH';

I’ve updated some info in countries, leaving the transaction open.

postgres=# begin transaction;
BEGIN
postgres=# select * from countries where country_code='CH';
 country_id | country_code |    name     |    official_name    | population | area_sq_km | latitude | longitude |   timezone    | region_id
------------+--------------+-------------+---------------------+------------+------------+----------+-----------+---------------+-----------
 CHE        | CH           | Switzerland | Swiss Confederation |    8293000 |   41277.00 | 47.00016 |   8.01427 | Europe/Zurich | EU
(1 row)

postgres=# update countries set population=population+1 where country_code='CH';
UPDATE 1
postgres=# select * from countries where country_code='CH';
 country_id | country_code |    name     |    official_name    | population | area_sq_km | latitude | longitude |   timezone    | region_id
------------+--------------+-------------+---------------------+------------+------------+----------+-----------+---------------+-----------
 CHE        | CH           | Switzerland | Swiss Confederation |    8293001 |   41277.00 | 47.00016 |   8.01427 | Europe/Zurich | EU
(1 row)

With this transaction still on-going, I’ll insert, from another session, a new child row for this parent value: another country in Switzerland

Session 2: insert a child

begin transaction;
select * from cities where country_id='CHE';
insert into cities values ('CHE1170', 'Aubonne', null, 2750, 'N', 46.49514, 6.39155, null, 'CHE');
commit;

I am able to commit my transaction without any locking problem.

Here is the output:

postgres=# begin;
BEGIN
postgres=# select * from cities where country_id='CHE';
 city_id | name | official_name | population | is_capital | latitude | longitude | timezone | country_id
---------+------+---------------+------------+------------+----------+-----------+----------+------------
 CHE0001 | Bern |               |     422000 | Y          | 46.94809 |   7.44744 |          | CHE
(1 row)

postgres=# insert into cities values ('CHE1170', 'Aubonne', null, 2750, 'N', 46.49514, 6.39155, null, 'CHE');
INSERT 0 1
postgres=# commit;
COMMIT

I was able to commit my transaction

Back to session 1

postgres=# select * from countries where country_code='CH';
 country_id | country_code |    name     |    official_name    | population | area_sq_km | latitude | longitude |   timezone    | region_id
------------+--------------+-------------+---------------------+------------+------------+----------+-----------+---------------+-----------
 CHE        | CH           | Switzerland | Swiss Confederation |    8293001 |   41277.00 | 47.00016 |   8.01427 | Europe/Zurich | EU
(1 row)

postgres=# select * from cities where country_id='CHE';
 city_id |  name   | official_name | population | is_capital | latitude | longitude | timezone | country_id
---------+---------+---------------+------------+------------+----------+-----------+----------+------------
 CHE0001 | Bern    |               |     422000 | Y          | 46.94809 |   7.44744 |          | CHE
 CHE1170 | Aubonne |               |       2750 | N          | 46.49514 |   6.39155 |          | CHE
(2 rows)

postgres=# commit;
COMMIT

In the first session, I can see immediately the changes because I’m in the default READ COMMITED isolation level. In SERIALIZABLE, I would have seen the new row only after I completed by transaction that started before this concurrent insert.

Oracle

Of course, Oracle can also run those transactions without any lock. There are no row level lock involved in those transactions except for the row that is modified by the transaction. But as long as the referenced key (the primary key of countries here) is not updated, there are no locks in the other table. Only if the key was updated, the index entry on the foreign key would have been “locked” to avoid inserting a row referencing a parent that will be removed.

I’ll not paste a demo here, as I have many ones available from this post: https://franckpachot.medium.com/oracle-table-lock-modes-83346ccf6a41. Oracle is very similar to PostgreSQL here. If you want to know the difference between Oracle and PostgreSQL about foreign keys, I’ve written about that on the CERN blog: https://db-blog.web.cern.ch/blog/franck-pachot/2018-09-unindexed-foreign-keys-oracle-and-postgresql

NoSQL

In NoSQL databases you don’t lock anything and you accept inconsistencies. Actually, all will depend on your data model. You may store a “country” item with a list of cities in a document store. Key: country id. Value: JSON with country attributes and list of cities with their attribute. Here either you lock in the same way MySQL does: don’t touch to the item while another one is modifying it. Or you can store those as multiple items (the “single table model” for DynamoDB for example) and then people can modify country and cities concurrently. They don’t lock each other but of course you may eventually live in a city which belongs to no country… This is the CAP theorem: scalability vs. consistency. You have this choice in your data model: either you cluster all items together in a single document, or you shard them within the datastore.

YugaByteDB

What about distributed databases? Having a foreign key referencing a parent in another node is a bit more complex because there is a compromise to define: wait on an internet latency, or raise an exception to re-try the operation. YugaByte DB is a NewSQL database which aims at full consistency, SQL and ACID, with the maximum scalability and availability possible.

Note that this is a new database, which has still a lot of work in progress in this area, all documented: https://docs.yugabyte.com/latest/architecture/transactions/explicit-locking, and implemented while the users are asking for it. So, if you read this several weeks after the publishing date… re-run the example and you may have good surprises.

I have a 3 nodes YugaByteDB cluster over 3 regions (I have a few Oracle Free Tier tenants, with free VMs always up).

{
curl -s https://raw.githubusercontent.com/gvenzl/sample-data/master/countries-cities-currencies/install.sql
} | /home/opc/yugabyte-2.5.2.0/bin/ysqlsh -h localhost -U yugabyte -d yugabyte -e

This has created and populated the table, auto-sharded into my 3 nodes:

The shards are called “tablets” here (they can additionally be replicated for HA and DR in but my replication factor is 1 here) and you can see that those PostgreSQL tables (there are multiple APIs in this YugaByte database) have leaders (would have followers with higher replication factor) in every node.

[opc@yb-fra-1 ~]$ /home/opc/yugabyte-2.5.2.0/bin/ysqlsh
ysqlsh (11.2-YB-2.5.2.0-b0)
Type "help" for help.

yugabyte=# begin transaction;
BEGIN

yugabyte=# select * from countries where country_code='CH';

 country_id | country_code |    name     |    official_name    | population | area_sq_km | latitude | longitude |   timezone    | region_id
------------+--------------+-------------+---------------------+------------+------------+----------+-----------+---------------+-----------
 CHE        | CH           | Switzerland | Swiss Confederation |    8293000 |   41277.00 | 47.00016 |   8.01427 | Europe/Zurich | EU
(1 row)

yugabyte=# update countries set population=population+1 where country_code='CH';
UPDATE 1

yugabyte=#

I have run the same as before: on one session updating the parent table.

[opc@yb-fra-3 ~]$ /home/opc/yugabyte-2.5.2.0/bin/ysqlsh
ysqlsh (11.2-YB-2.5.2.0-b0)
Type "help" for help.

yugabyte=# begin transaction;
BEGIN

yugabyte=# select * from cities where country_id='CHE';

 city_id | name | official_name | population | is_capital | latitude | longitude | timezone | country_id
---------+------+---------------+------------+------------+----------+-----------+----------+------------
 CHE0001 | Bern |               |     422000 | Y          | 46.94809 |   7.44744 |          | CHE
(1 row)

yugabyte=# insert into cities values ('CHE1170', 'Aubonne', null, 2750, 'N', 46.49514, 6.39155, null, 'CHE');

ERROR:  Operation failed. Try again.: d993bd6d-2f76-40ea-9475-84d09e5e0438 Conflicts with higher priority transaction: efd1aaaf-12f9-4819-9e0c-41ee9c6cfb7b

yugabyte=# commit;
ROLLBACK

When inserting a new child, I got a failure because optimistic locking is used there. We don’t wait but serialization conflicts have to cancel one transaction.

Note that it could be one or the other session that is canceled. Running the same example again I was able to insert and commit a child:

yugabyte=# insert into cities values ('CHE1170', 'Aubonne', null, 2750, 'N', 46.49514, 6.39155, null, 'CHE');
INSERT 0 1

yugabyte=# commit;
COMMIT

This is successful but now the session 1 transaction is in conflict.

select * from countries where country_code='CH';

 country_id | country_code |    name     |    official_name    | population | area_sq_km | latitude | longitude |   timezone    | region_id
------------+--------------+-------------+---------------------+------------+------------+----------+-----------+---------------+-----------
 CHE        | CH           | Switzerland | Swiss Confederation |    8293001 |   41277.00 | 47.00016 |   8.01427 | Europe/Zurich | EU
(1 row)

yugabyte=# commit;

ERROR:  Operation expired: Transaction expired or aborted by a conflict: 40001

yugabyte=# select * from countries where country_code='CH';

 country_id | country_code |    name     |    official_name    | population | area_sq_km | latitude | longitude |   timezone    | region_id
------------+--------------+-------------+---------------------+------------+------------+----------+-----------+---------------+-----------
 CHE        | CH           | Switzerland | Swiss Confederation |    8293000 |   41277.00 | 47.00016 |   8.01427 | Europe/Zurich | EU
(1 row)

The transaction started in session 1 cannot be completed and must be re-tried.

This is a general idea with distributed databases, SQL or NoSQL, optimistic locking is often preferred for scalability: better be ready to re-retry in the rare case of conflict rather than waiting for lock acquisition on other nodes. But I mentioned that one or the other transaction was cancelled. Which one? at random? Yes, by default each transaction has a random priority assigned. However, when I test or demo the behavior I want predictable results. This is possible by reducing the range of random priority within non-overlapping bounds.

The defaults are between 0 and 1:

yugabyte=# show yb_transaction_priority_upper_bound;
 yb_transaction_priority_upper_bound
-------------------------------------
 1
(1 row)

yugabyte=# show yb_transaction_priority_lower_bound;
 yb_transaction_priority_lower_bound
-------------------------------------
 0
(1 row)

Now, if I set the session 1 in the lower range (yb_transaction_priority_lower_bound=0, yb_transaction_priority_upper_bound=0.4) and the session 2 in the higher range (yb_transaction_priority_lower_bound=0.6, yb_transaction_priority_upper_bound=1), I know that the session 1 will have its transaction aborted on conflict (look at the timestamps, or colors if you can see them: I’ve run blue first, with low priority, then green with higher priority in the bottom session, then red, back to first session where it fails):

But if I set the session 1 in the higher range (yb_transaction_priority_lower_bound=0.6, yb_transaction_priority_upper_bound=1) and the session 2 in the lower range (yb_transaction_priority_lower_bound=0, yb_transaction_priority_upper_bound=0.4) I know that the session 2 will have its transaction failed on conflict (I started with green here in higher priority session 1, then blue and red in the bottom session where it failed):

Again, this is a choice: optimistic locking. Better retry sometimes than locking always.

Data Model

However, even if the database can implement locking with good efficiency, you should think about the data model. Forget about normal forms here. Think of tables as business entities where attributes are tightly coupled together, but coupling between entities is just a possibility for some use cases. When looking at the domain model, CITY is an entity, and COUNTRY is another. However, it may require some transformation between the domain model and the implementation. There is the COUNTRY as a code, with a name and which is an aggregation of CITY. And it may be a different entity from the country as location where people live, with the population number that I’ve updated. Maybe the POPULATION attribute I’ve been updating belongs this other COUNTRY_POPULATION table. Don’t worry about joins: join can scale in a RDBMS with purpose-built joins algorithms. And, anyway, maybe one day this table with have temporality added because population changes and history is interesting to keep. And with this data model, whatever the locking mechanisms, you can update the population and and cities without blocking each-others. That’s how I think about normalization: business meaning, coupling, cardinalities, evolution…

Let me show that in my YugaByte sessions:

YSQL1 20:26:58 create table COUNTRY_POPULATION as select * from COUNTRIES;
SELECT 196
YSQL1 20:27:06 alter table COUNTRY_POPULATION add foreign key (COUNTRY_ID) references COUNTRIES;
ALTER TABLE
YSQL1 20:27:55

I have created another table for the population info, with a foreign key to COUNTRIES. Of course, I should remove the columns from the tables, keeping only the foreign key, but that doesn’t change my example and I keep it simple.

The chronology here is: green -> blue -> yellow. Here, the consistency of data is enforced by Foreign Key referential integrity, scalability is ensured by sharding across nodes over the internet, and there’s no lock conflict thanks to my data model.

There’s no reason to remove foreign keys here, thanks to correct relational data modeling, and efficient handling even in a the distributed database. If you can’t rely on the database to handle that, you need lot of additional code, complex testing for race condition and failure scenario, or you risk hard to recover data inconsistency.

L’article Foreign Keys in MySQL, SQL, NoSQL, NewSQL est apparu en premier sur dbi Blog.

The problem

I was doing some tests on the new MySQL Database Service and during the setup I decided to generate my ssh keys via the OCI console:

When I tried to connect via PuTTY or MobaXterm to my compute instance using the opc account and my private key (generated previously), I got the following error:

Looking at the keys generated via the Oracle Cloud console, I saw that they were defined in the following format:

The solution

Actually I don’t work directly on a Linux system. So I need to convert my private key if I want to make it usable via my connection tools.
First step is to transform it to RSA format. I can do it using OpenSSL:

# openssl rsa -in ssh-key-2020-11-24.key -out ssh-key-2020-11-24.rsa

Second and last step is to convert it to ppk format. I can do it using PuTTYgen.
I load the private key:

I filter on all files types:

I select my RSA key and I click on Open:

I click on Ok on the following message:

and then on Save private key:

So I save the key with a ppk format:

Tests

I can now use my private key to connect to my OCI compute instance via PuTTY:

or MobaXterm:

Hope this can help you!

L’article Convert private key generated via OCI Console to ppk est apparu en premier sur dbi Blog.