Out of the 300 features encapsulated in the new Oracle 23c database, I’ve chosen to write during the Oracle CloudWord a brief blog post about a feature that particularly caught my attention: Automatic Transaction Rollback.

Starting from version 23c, Oracle enables automatic transaction rollback with parameters to manage this functionality. Through this blog post I’ll demonstrate, with a practical example how this functionality works.

Use case scenario

Let’s consider the following scenario: two developers are working on the table “employees” , and a critical batch process runs every 30 minutes on this same table.

The first developer (Dev1) is performing an update operation that takes some time. Instead of waiting for the update to complete and deciding whether to commit or rollback the transaction, Dev1 decides to go to the cafeteria.

The second developer (Dev2) is also working on the same project and needs to update the same table shortly after Dev1 started his update. However, because of the lock created by Dev1’s update, Dev2 is unable to execute his transaction. Becoming impatient while waiting at the laptop, Dev2 decides to leave for home, leaving the transaction pending.

After a few minutes, the critical batch process (executed by “financial_app”) starts, but it too becomes stuck due to the ongoing update on the “employees” table.

Transaction Rollback with Oracle 23c

A situation like this can now be managed using Automatic Transaction Rollback. To set up transaction rollback, we need to consider two (or three) parameters:

1. Transaction priority (txn_priority), which can be set on session level to low, medium or high.
2. Time a transaction holding row lock can be automatically rolled back. This setting can be set in the system parameters (TXN_AUTO_ROLLBACK_HIGH_PRIORITY_WAIT_TARGET & TXN_AUTO_ROLLBACK_MEDIUM_PRIORITY_WAIT_TARGET)

Transaction priority

The priority can be changed on session level using the below syntax:

ALTER SESSION SET "txn_priority" = "HIGH";

As stated in the Oracle documentation:

• If a HIGH priority transaction is blocked for a row lock, Oracle database can roll back the transaction that is holding the row lock only if the holder is LOW or MEDIUM priority.
• If a MEDIUM priority transaction is blocked for a row lock, Oracle database can roll back the transaction that is holding the row lock only if the holder is LOW priority.
• If a LOW priority transaction is blocked for a row lock, Oracle database will not attempt to roll back the transaction holding the row lock irrespective of its priority.
• Oracle database never rolls back a HIGH priority transaction.

Time a transaction holding row lock can be automatically rolled back

There are two system parameters which define the time (in seconds) after a transaction holding row lock can be automatically rolled back:

1. TXN_AUTO_ROLLBACK_HIGH_PRIORITY_WAIT_TARGET 
2. TXN_AUTO_ROLLBACK_MEDIUM_PRIORITY_WAIT_TARGET 

There is no low priority wait target parameter provided since Oracle database doesn’t roll back a blocker transaction if waiter’s priority is LOW.

These system parameters can be defined using the below command (example for txn_auto_rollback_high_priority_wait_target):

ALTER SYSTEM SET txn_auto_rollback_high_priority_wait_target = 15;

Preparation of the environnement to run our example

Let’s revisit our example with two developers (dev1 and dev2) and one critical application (financial_app) and break down this scenario step by step:

Users creation

This step doesn’t add much to my example, except that it allows me to demonstrate the use of another new and interesting feature: the DB_DEVELOPER_ROLE. This role provides the fundamental roles and privileges that Oracle believes are essential for a database developer.

SQL> create user dev1 identified by dev1;
SQL> create user dev2 identified by dev2;
SQL> create user financial_app identified by financial_app;
SQL> alter user financial_app quota 128M on users;
SQL> grant DB_DEVELOPER_ROLE to dev1, dev2, financial_app;

… well let’s say two others features with the “grant on schema” features (schema level privilege):

SQL> grant select any table on schema financial_app to dev1, dev2;
SQL> grant insert any table on schema financial_app to dev1, dev2;
SQL> grant update any table on schema financial_app to dev1, dev2;

Table creation

Let’s create a very basic employees table:

SQL> CREATE TABLE financial_app.employees ( employee_id number(10) NOT NULL, employee_name varchar2(50) NOT NULL, city varchar2(50) );

We will now insert few rows:

SQL> insert into financial_app.employees (employee_id, employee_name, city) values (1, 'Westermann', 'Basel'), (2, 'Wisson', 'Nidau'), (3, 'Usai', 'Sutz'), (4, 'Berbier', 'Zurich');

Priority wait definition

For the sake of this example, we will assign different wait target to “Medium” and “High” priority transactions.

SQL> ALTER SYSTEM SET txn_auto_rollback_high_priority_wait_target = 15;
SQL> ALTER SYSTEM SET txn_auto_rollback_medium_priority_wait_target = 30;

Let’s run our example

As mentionned in our introduction we have three users with three different sessions (Dev1, Dev2 and financial_app). Dev1 and Dev2 will both update employees table and won’t commit nor rollback their transaction. Each of these three users will have a different priority:

Dev1 – priority low

[oracle@grs-oraclecloudinstance bin]$ ./sqlplus dev1/dev1
SQL> ALTER SESSION SET "txn_priority" = "LOW";

Dev2 – priority medium

[oracle@grs-oraclecloudinstance bin]$ ./sqlplus dev2/dev2
SQL> ALTER SESSION SET "txn_priority" = "MEDIUM";

financial_app – priority high

[oracle@grs-oraclecloudinstance bin]$ ./sqlplus financial_app/financial_app
SQL> ALTER SESSION SET "txn_priority" = "HIGH";

Update execution

Dev 1

First of all we have Dev1 who runs his update on employees table. In this example he only updates one row and it’s obviously very fast but anyway he do not commit his transaction.

SQL> UPDATE financial_app.employees set employee_name = 'Steulet' where employee_ID=1;
1 row updated.

SQL> SELECT TO_CHAR(SYSDATE, 'MM-DD-YYYY HH24:MI:SS') FROM dual;
TO_CHAR(SYSDATE,'MM
-------------------
09-21-2023 01:08:30

Dev2

5 second later, we have Dev2 who runs his update on employees table but as mentionned before Dev1 forgot to commit his transaction.

SQL> SELECT TO_CHAR(SYSDATE, 'MM-DD-YYYY HH24:MI:SS') FROM dual;
TO_CHAR(SYSDATE,'MM
-------------------
09-21-2023 01:08:35

SQL> UPDATE financial_app.employees set employee_name = 'Schweitzer' where employee_ID=1;

Here the transaction is stuck due to the row lock but after 30 seconds we get the below confirmation:

1 row updated.

SQL> SELECT TO_CHAR(SYSDATE, 'MM-DD-YYYY HH24:MI:SS') FROM dual;
TO_CHAR(SYSDATE,'MM
-------------------
09-21-2023 01:09:05

Financial_app

The second developer, dev2, also did not commit his transaction but the financial_app runs his update at 01:10:16:

SQL> SELECT TO_CHAR(SYSDATE, 'MM-DD-YYYY HH24:MI:SS') FROM dual;
TO_CHAR(SYSDATE,'MM
-------------------
09-21-2023 01:10:16

SQL> Update financial_app.employees set employee_name = '*';

As before the transaction his stuck due to the row lock but after 15 seconds we get the following confirmation:

1 row updated.

SQL> SELECT TO_CHAR(SYSDATE, 'MM-DD-YYYY HH24:MI:SS') FROM dual;
TO_CHAR(SYSDATE,'MM
-------------------
09-21-2023 01:10:31

What happened?

As often, a picture is worth a thousand words. The image below summarizes what happened with these three sessions.

User Dev 1 runs his update at 01:08:30, he immediately received the confirmation that his row was updated. However, he did not commit his transaction, which resulted in a lock on the employees’ table.

5 seconds later at 01:08:35, Dev2 also performed an update on the same table. He did not receive an immediate confirmation of his update. However, after 30 seconds, at 01:09:05, the confirmation arrived. This eventually caused the session of Dev1 to end with a rollback of his transaction. Dev 2, did not commit his transaction neither, which also resulted in a lock on the employee’s table.

Few seconds later at 01:10:16, financial_app runs his batch on employees table. As for Dev2, the batch was not able to execute immediately the update. However after 15 seconds, the confirmation arrived. This caused the session of Dev2 to end with a rollback of his transaction.

The content of the employees’ table at 01:10:31 in the financial_app’s session is as follows:

SQL> set linesize 200
SQL> select * from financial_app.employees;
EMPLOYEE_ID EMPLOYEE_NAME             CITY
----------- ------------------------- ---------------------
          1 *                         Basel
          2 *                         Nidau
          3 *                         Sutz
          4 *                         Zurich

L’article Managing transaction priority in Oracle 23c est apparu en premier sur dbi Blog.

Larry Ellison kicked off his keynote by delving into Artificial Intelligence, particularly focusing on ChatGPT. The phrase is sure to become famous “About a year ago OpenAI demonstrated ChatGPT 3.5 and much to the shock of the people who developed ChatGPT… the baby talks. Seriously the developers did not expect that“.

Larry then revisited OpenAI’s history, explaining that it was developed a decade ago. He notablity explained that they constructed an enormous neural network composed of billions of artificial neurons. After building it, they extensively trained it on a vast amount of data, including the entire public Internet. However, the underlying concept remained consistent with previous iterations. What set ChatGPT 3.5 apart was its larger neural network and broader training data. Due to this increase in scale, when prompted with questions, ChatGPT unexpectedly provided answers and engaged in conversation. This revelation surprised the ChatGPT team, AI professionals, and the world at large, capturing our collective imagination.

However, as Larry mentioned, most cool tech doesn’t come without risks. Everyone wants to know what comes next. “What comes next is a wordwide race to build what comes next, to build better AI“. While better features will undoubtedly emerge, they also bring new risks, like any new technology. “From early innovations like fire, which could be misused, to later technologies such as nuclear power, all new technology carries potential for misuse“.

As stated in Larry’s slides: “Is Generative AI the Most Important New Computer Technology Ever? Probably!”

To the question what does the future hold? “Complete self-driving cars within the next 12 months from Tesla, computer-designed antiviral drugs, and much more.“

Oracle Cloud Best Cloud for training Generative AI Models

Larry mentionned that Oracle Gen2 Cloud stands out from other clouds in several significant ways. According to him, one of the key distinctions is the network that Oracle employs to interconnect its computers. He refers here about the RDMA (Remote Direct Memory Access) Network which connects NVIDIA GPUS into Huge Superclusters. This network grants Oracle the ability to transfer large volumes of data between computers at incredible speeds, surpassing conventional networks. This remarkable speed enhances Oracle Cloud’s capacity for rapidly training AI models.

In his own words: “We are much faster and many times less expensive than the other clouds for training AI Models“

NVIDIA Supercomputer

Larry Ellison then came back on NVIDIA’s announcement regarding the construction of the world’s fastest AI supercomputer, leveraging the NVIDIA H100 GPU and the Oracle Cloud RDMA Network. He emphasized that while the processing speed of data is crucial, the ability to move data quickly between computers is equally vital; otherwise, processors remain idle. Oracle’s exceptional data transfer speed advantage stems from their early adoption of this technology for their own database technology, effectively solving the data movement challenge.

Generative AI is Transformational

Larry explained that Generative AI is fundamentally changing how Oracle (will) build and run applications. He provided the following examples:

• Improved APEX Application Generator: No Code – not – Low code
• Improved Autonomous Elastic Database: No Human Labor – No Human Error
• New Object Relational Database: Generate Relational Schema from JSON documents
• New Oracle Vector Database: Specializes Generative LLMS for Medicine, Law…
• New Cloud Data Intelligence Platform: Population Scale Data Trains Medical AI Models

Apex for all future Oracle Applications development?

According to Oracle’s CTO and co-founder, most new Oracle applications will now be generated with APEX, including Cerner New Millennium, Fusion Marketing, Banking, Retail, Hospitality, and more. APEX enables development teams to be streamlined, significantly reduces development time, and ensures more secure and reliable results. While existing Java projects at Oracle will continue, no new Oracle products will be developed in Java, despite Oracle’s history of using Java for development.

Additionally, all Oracle new or strategic applications will now run on Autonomous Database, typically Fusion & NetSuite will move to Autonomous Elastic Database. Why this decision?

“If there is no human labor, there is no human error… the only way to build a secure system is to eliminate human error.”

Larry then discussed the utilization of AI in the healthcare sector, including medical AI models, and the methods for collecting patient data using Cloud IoT devices such as blood pressure monitors, pulse oximeters, smartwatches, and more.

AI can transform other industries

Following the previous examples that were given and aside from medicine, there are other industries that can be transformed by Artificial Intelligence. The CTO and co founder provided us with a compelling example from the agriculture sector, specifically Greenhouse Agriculture Robots. These technologies offer various advantages, including:

• Grow using 98% Less water
• Grow using 90% less land
• Grow new Population Centers
• Daily Harvest with Same Day Delivery
• Better Job Security and Job Quality

Other examples were shared, such as the current police cars. This also presented the opportunity to give us a glimpse of the future police cars (Larry and Elon’s personal favorites ). A few key points about these cars: “They are safe, exceptionally fast, and have stainless steel bodies.“… well I would say not a surprise.

MultiCloud: Microsoft & NRI

Last topic that Larry addressed was his visit to Redmond last week where he met with Microsoft’s CEO, Satya Nadella. During their meeting, they reached a consensus on significant ideas: “Clouds should be open” and “Clouds should be interconnected“. Many customers currently use multiple cloud services, including AWS, Azure, Google, Salesforce, and Oracle.

Customers should find it easy to utilize various cloud services and seamlessly interconnect them, regardless of the provider. The concept revolves around ensuring that all these clouds can effectively interoperate.

Both clouds are open and both cloud providers would like to avoid Egress fees on customers data. To the question “Is it a complete Oracle Cloud Inside the Azure Datacenter the response is Yes !“

L’article Larry Ellison – Oracle’s Vision for the future est apparu en premier sur dbi Blog.

It is true that it’s a somewhat daring title, and it’s unlikely that SQL firewall has changed anything about the attack that several casinos in Las Vegas suffered last week. However, the opportunity was too good to pass up on this topic.

As I write this blog, Las Vegas casinos are experiencing a series of cyberattacks. It turns out that at the same time, OracleCloud World is taking place in Las Vegas, and one of its flagship products, the database, offers a new feature in its 23c version called ‘SQL firewall.’ So, the opportunity was too good not to write a short blog about this new functionality. I would like to emphasize here that this is only a very brief overview of all the features of DBMS_SQL_FIREWALL. Oracle provides extensive documentation on this package/feature, and you will find several blogs on the internet that go beyond the tests below (have a look in my conclusion)

How does SQL Firewall work?

The principle of the package is very simple and is illustrated through the image below, which has been taken from the Oracle documentation:

1. First, we capture the standard activity of an application (Capture SQL Activities). Naturally, the capture must run for a sufficient amount of time to cover all specific scenarios.
2. By examining this activity (Review), we ensure that there is nothing abnormal.
3. We generate the list (Generate Allow-List) of allowed execution contexts. This list can be modified if necessary by adding or removing specific executions using the following package procedures: DBMS_SQL_FIREWALL.ADD_ALLOWED_CONTEXT, DBMS_SQL_FIREWALL.DELETE_ALLOWED_CONTEXT, and DBMS_SQL_FIREWALL.DELETE_ALLOWED_SQL.
4. We enable the list (Enable Allow-List).
5. We monitor (Monitor) the operation using the DBA_SQL_FIREWALL_VIOLATIONS dictionary view.

Example

For this example I will need two users:

• schema_owner who is the owner of the tables & views of the application
• app_user who needs to access to the application

We could use a specific user to manage database firewall by providing him the sql_firewall_admin role but as stated in my introduction I’ll keep this example very short and simple.

Users creation

Creation of the user schema_owner:

SQL> create user schema_owner identified by schema_owner quota unlimited on users;
grant connect,resource to schema_owner;
grant create session grant any privilege to schema_owner;
User created.

Creation of the user app_user:

SQL> create user app_user identified by app_user;
grant create session to app_user;
grant select any table on schema schema_owner to app_user;
User created.

Objects creation

Now we will create few objects in our schema_owner schema:

SQL> CREATE TABLE schema_owner.customers
( customer_id number(10) NOT NULL,
customer_name varchar2(50) NOT NULL,
city varchar2(50),
credit_card number(16),
CONSTRAINT customers_pk PRIMARY KEY (customer_id)
);
Table created.

SQL> create view schema_owner.non_sensitive_view as select customer_name, city from schema_owner.customers;

View created.

Data insertion

We will now insert data in our table:

SQL> insert into schema_owner.customers values (001,'steulet','Delemont',00001111222233334444);
insert into schema_owner.customers values (002,'errard','Courrendlin',2222333344445555);
insert into schema_owner.customers values (003,'frund','Porrendutry',7777888899990000)
1 row created.

Start the capture of the statements (Capture SQL Activities)

Before being able to capture the statements, we have to be sure that SQL Firewall is enabled:

SQL> select status
from dba_sql_firewall_status; 
STATUS
------------
DISABLED

SQL> exec dbms_sql_firewall.enable;

PL/SQL procedure successfully completed.

SQL> select status
from dba_sql_firewall_status;
STATUS
------------
Enable

We can now start the capure process of the standard context for the user app_user using the below procedure:

SQL> begin
dbms_sql_firewall.create_capture (
username => 'app_user',
top_level_only => false,
start_capture => true);
end;
 
PL/SQL procedure successfully completed.

Let’s run the application

We now need to run the application for the necessary duration in order to collect all user context information related to those who will be querying the application. We will greatly simplify this phase by simply performing a select on the view we created earlier. This select will be done in a separate session using the application user login & password:

[oracle@grs-oraclecloudinstance bin]$ ./sqlplus app_user/app_user@free

SQL*Plus: Release 23.0.0.0.0 - Developer-Release on Mon Sep 18 03:16:56 2023
Version 23.2.0.0.0Copyright (c) 1982, 2023, Oracle. All rights reserved.Connected to:
Oracle Database 23c Free, Release 23.0.0.0.0 - Developer-Release
Version 23.2.0.0.0
SQL> set linesize 200 pagesize 40
SQL> select * from schema_owner.non_sensitive_view;

CUSTOMER_NAME                                      CITY
-------------------------------------------------- ---------------------------------
steulet                                            Delemont
errard                                             Courrendlin

Let’s stop the capture

We can admit that the application ran for the necessary duration in order to catch all the contextual information and we stop the capture:

SQL> exec dbms_sql_firewall.stop_capture('app_user');
PL/SQL procedure successfully completed.

Let’s have a look on what we captured (Review)

When querying dba_sql_fiewall_capture_logs, one can view the logs as demonstrated below:

SQL> column commande_type format a9
column current_user format a10
column client_program format a20
column os_user format a8
column ip_address format a10
column sql_text format a30
select command_type, current_user, client_program, os_user, ip_address, sql_text from dba_sql_firewall_capture_logs where username='APP_USER';

COMMAND_TYPE CURRENT_US CLIENT_PROGRAM       OS_USER  IP_ADDRESS SQL_TEXT
------------ ---------- -------------------- -------- ---------- ------------------------------
SELECT       APP_USER   sqlplus@grs-oraclecl oracle   10.0.0.79  SELECT * FROM SCHEMA_OWNER.NON
                        oudinstance (TNS V1-                     _SENSITIVE_VIEW
                        V3)

SELECT       APP_USER   sqlplus@grs-oraclecl oracle   Local      SELECT DECODE (USER,:"SYS_B_0"
                        oudinstance (TNS V1-                     ,XS_SYS_CONTEXT (:"SYS_B_1",:"
                        V3)                                      SYS_B_2"),USER) FROM SYS.DUAL

SELECT       APP_USER   sqlplus@grs-oraclecl oracle   10.0.0.79  SELECT DECODE (USER,:"SYS_B_0"
                        oudinstance (TNS V1-                     ,XS_SYS_CONTEXT (:"SYS_B_1",:"
                        V3)                                      SYS_B_2"),USER) FROM SYS.DUAL
  

To read the output more easily you will find a screenshot below:

Let’s generate the allow list (Generate Allow-List)

Using DBMS_SQL_FIREWALL.GENERATE_ALLOW_LIST we generate the allow_list:

SQL> exec DBMS_SQL_FIREWALL.GENERATE_ALLOW_LIST ('APP_USER');
PL/SQL procedure successfully completed.

Let’s enable protect our database (Enable Allow-List)

Using the below procedure we will activate the allow list and block violations:

SQL> begin
dbms_sql_firewall.enable_allow_list (
username => 'app_user',
enforce => dbms_sql_firewall.enforce_all,
block => true);
end;
/ 

PL/SQL procedure successfully completed.

What about a test?

Let’s run a query that hasn’t been run before. We get a SQL Firewall violation:

[oracle@grs-oraclecloudinstance bin]$ ./sqlplus app_user/app_user@free

SQL> select credit_card from schema_owner.customers;
select credit_card from schema_owner.customers
*
ERROR at line 1:
ORA-47605: SQL Firewall violation

What can we find in the log?

Inspecting the log we can have some details regarding the rule violation:

SQL> select sql_text,
firewall_action,
ip_address,
cause,
occurred_at
from dba_sql_firewall_violations;

SQL_TEXT                       FIREWAL IP_ADDRESS CAUSE                OCCURRED_AT
------------------------------ ------- ---------- -------------------- -------------
SELECT CREDIT_CARD FROM SCHEMA Blocked 10.0.0.79  SQL violation        18-SEP-23 04.08.54.371122 AM +00:00
_OWNER.CUSTOMERS

To read the output more easily you will find a screenshot below:

Conclusion

Oracle provides a series of tools to control data access. SQL Firewall is added to the list of these tools, such as DBM_CRYPTO, TDE, Oracle Virtual Private Database, to name just a few. SQL Firewall is used in this blog to block queries, but it can also be configured for query monitoring only. It’s also interesting to note that we can use Oracle Data Pump to export and import SQL Firewall captures and allow-list metadata. Of course on large volume of Data SQL Firewall can have an impact on performance (I didn’t test it). In such a context Oracle recommends the two below measures:

• Allocate at least an additional 2G to the LARGE_POOL_SIZE parameter setting, on top of the existing LARGE_POOL_SIZE requirement.
• Resize the SGA_TARGET parameter setting to include this additional requirement. Ensure that the final size is 8G or more.

Among the different blogs that address this topic we can find:

• 23c Oracle Firewall from Rob Watson
• SQL Firewall Oracle 23c from Pete Finnigan
• SQL Firewall in Oracle Database 23c from Oracle-Base (Tim)
  

L’article Oracle to the rescue of MGM with SQL Firewall? est apparu en premier sur dbi Blog.

Oracle CloudWorld 2023 will start next week in Las Vegas, and I’m happy to participate in this worldwide event. As a Swiss citizen, when I think of Las Vegas, the first image that comes to my mind is that of dazzling lights, opulent casinos, and dancing fountains. A city that never sleeps, where excess seems to be the norm. However, behind this glittering facade lies a surprising reality: Las Vegas appears to be a leader in many sustainable initiatives and a model of responsible water management. It’s perhaps also because of this that a company such as Oracle decided to organize its most famous event in this city.

Environmental Savior or Sinful Excess?

While it won’t be my first visit to Sin City, as Las Vegas hosts several events like CES and is a popular destination for those exploring the West Coast on road trips, each visit brings its own unique experiences. On my previous trips to Las Vegas, what truly captivated my attention was how a city nestled in the heart of the desert could orchestrate mesmerizing water spectacles, such as the renowned ones adorning the famous Las Vegas Strip.

As a company that has dedicated significant efforts to sustainability initiatives and proudly holds Ecovadis certification, I felt compelled to delve deeper into the question of how this desert city manages its energy, and its precious water resources. To craft this article, I had the privilege of engaging with Jace Radke, the Public & Media Relations Supervisor in Las Vegas as well as Mack Branson, who represents both the Southern Nevada Water Authority (SNWA) and the Las Vegas Valley Water District. I thank them for generously sharing their time and expertise, which greatly enriched the content of this article.

World-class in terms of sustainability

As stated by Jace Radke, “More than 2.1 million people reside in Southern Nevada and 40 million people from around the world travel to Las Vegas annually. Many know Las Vegas as a world-renowned entertainment and tourist destination, however, very few of them know that Las Vegas is also world-class when it comes to being green and sustainable.” Just to provide few facts and figures that attests the efforts of Las Vegas in terms of sustainability (a more exhaustive list can be found here: www.lasvegasnevada.gov/sustainability) :

• Certifications such as LEED (Leadership in Energy and Environmental Design) certified. buildings, including Las Vegas City Hall that opened in 2012.
• Investment in hundreds of miles of bike lanes.
• Completion of more than six megawatts of solar.
• Replacement of 42,000 streetlights with LED lighting.
• Reduced annual water consumption by 256 million gallons from 2008 level.
• Every public building, park and streetlight is served by renewable energy.
• Several years ago, the largest rooftop solar array in the country went online at Mandalay Bay, with 26,000 panels covering 28 acres.

Many other initiatives will be undertaken. For example, the city plans to plant 60’000 new trees to help mitigate the heat island impacts in the city and lower temperatures. This initiative is part of the city’s 2050 Master Plan: https://www.lasvegasnevada.gov/Business/Planning-Zoning/Master-Plan

These measures have been awarded. In 2014, the city was honored with the US Conference of Mayors’ Climate Protection Award in Dallas. Such initiatives also led to a reduction in annual energy costs, decreasing from a peak total of $15 million in 2008 to less than $10 million per year.

However, despite these initiatives, Jace Radke remains aware of the ongoing challenges that pose a threat to community well-being: “Many communities, like Las Vegas, are facing issues more heat waves, flooding, or drought – an issue they face together with cities across the Southwest. Of course, water is clearly a big part of our story”.

While everything may not be perfect in Vegas in terms of sustainability, there is still room for improvement. However, when we examine the 2050 Master Plan, it becomes evident that the authorities are well aware of this potential for enhancement.

Water savings of 48’425 Olympic-size swimming pools

Before delving into water management, it’s crucial to establish a rough idea of the proportions we are discussing. As mentioned by Jace Radke and confirmed by Mack Branson, “while many look at Las Vegas’s growth and water use at casinos, golf courses and fountains as the cause of declining water levels in Lake Mead, the fact is that Las Vegas uses less than 4% of Colorado River water. Additionally, it’s important to mention that the resort industry only uses 3% of all water used in Las Vegas.”.

To maintain the focus on proportions, consider that the Southern Nevada region receives 100 millimeters of precipitation per year, while a country like Switzerland receives between 1000 to 1500 millimeters of precipitation per year (according to meteosuisse.admin.ch). Naturally, these conditions have prompted the Southern Nevada Water Authority to implement certain restrictions, such as:

• Replacing useless grass
• Limiting water gardens
• Prohibiting installation of new gras
• Limiting residential pool sizes
• Moratorium on evaporative cooling
• Annual water budgeting for Golf courses

As explained by Mack Branson, some strong measures will be enforced: “by the end of 2026, it will be illegal to irrigate grass which is considered decorative grass. This measure is due to the fact that grass needs approximately 3000 liters of water per square meters (73 Gallon per square foot). The reason of this measure is very simple, water that we use outdoor can be used only one time which is not the case of water used indoor because we can recycle it.“

Due to all these measures, Southern Nevada Authorities estimates that: the Southern Nevada’s consumptive water use declined by approximately 32 billion gallons (121.25 trillion liters) between 2002 and 2022, despite the influx of approximately 750,000 new residents and nearly 40 million annual visitors during that span. (Conservation fact sheet – Comprehensive 2023.pdf edited by Southern Nevada Water Authority) which represents approximately 48’425 Olympic size swimming pools.

Lake Mead, a challenge that goes beyond the borders of Las Vegas

As we all witnessed in recent years, the water level of Lake Mead significantly declined. Fortunately, the water level has rebounded in the past months. Nevertheless, to confront the challenge of declining mean water levels, it’s important to take into consideration that Las Vegas uses only 4% of the Colorado River water. Therefore, additional efforts are required beyond Las Vegas and Nevada.

As mentioned by Mack Branson “Our community recycles and returns nearly every drop of indoor water use back to Lake Mead, which provides 90 percent of Southern Nevada water supply. Every liter returned allows us to tank another out of the lake, treat it to drinking water standards, and deliver it to the community. We are the only community in the U.S. that we know of that recycles all of its indoor water use.”

However, Mack Branson is aware that the challenges related to Lake Mead do not stop at the borders of Las Vegas.: “It’s going to require every sector in every state that uses Colorado River water to use less. The largest water user of the Colorado River by sector is agriculture, which is beneficial to us in our nation’s food supply. However, a significant portion of that agricultural water use still employs inefficient irrigation techniques and produces a lot of feed for cattle (alfalfa). It appears that alfalfa has become one of the dominant water users in the agricultural sector. While it’s essential to support the dairy industry, we also observe a substantial portion of this alfalfa being exported out of the United States. Thus, we are effectively exporting Colorado River water out of the United States.

Intake 3, a drought insurance

In response to unprecedented drought conditions and for the long-term preservation of Southern Nevada’s primary water storage reservoir, Lake Mead, the Southern Nevada Water Authority built a third drinking water intake system (intake 3) capable of extracting Colorado River water when the lake’s elevation is below 1,000 feet (~304 meters) above sea level.

In 2008, construction of the intake 3 commenced, involving the use of a 24-foot (~7,3 meters) diameter tunnel boring machine to excavate a tunnel beneath Lake Mead. This tunnel was subsequently connected to an intake structure anchored to the lakebed with over 1,000 truckloads of concrete.

Currently, Lake Mead stands at a height of 1,066 feet (325 meters), as indicated by: https://mead.uslakes.info/level.asp. This means it is below the level of Intake Number 1. Consequently, Pumping Stations Number 2 and Number 3 are in operation. In a few months, when the Lake Mead elevation rises above Intake Number 1, Pumping Station 3 will be deactivated to conserve energy. This decision is driven by the greater energy consumption of Pumping Station Number 3, attributable to the depth of Intake Number 3.

Conclusion

The challenges encountered by Las Vegas, and more globally South-West, are challenges that will impact other regions around the world in the future. Behavior adaptations such as the replacement of grass, water recycling techniques, strategies to reduce water evaporation, and environmental certifications for hotels are all practices that are expected to become more widespread in the coming years.

Undoubtedly, Las Vegas is a city with high energy demands, and there is still much work to be done. However, over the past several years, awareness has been steadily increasing, giving rise to promising ideas and a genuine commitment to making meaningful improvements.

Climate change is affecting all of us. We need to adapt to build resilience. Climate change in the Southwestern United States will result in a warmer and drier future and if we do not admit that degrowth can be a solution, people will have to be innovative to face these challenges.

L’article Las Vegas, from sin city to green city? est apparu en premier sur dbi Blog.

Introduction

This blog is the third chapter related to deploying a MySQL infrastructure on the Azure cloud. In addition to performance, we should indeed also consider backup and restore capabilities. The objective of this blog is to present the main backup and restore possibilities offered by Azure through a simple example and show a second backup/restore possibility using MySQL Shell dump utilities.

Backup window and mechanism

Flexible Azure for MySQL generates by default a 7 days server backup retention period. This retention period can be extended up to 35 days or shorten to 1 day. In addition we can decide if you want to have a Geo Redundant backup storage. By default the backups are locally redundant.

It’s important to understand that Azure makes backup of the whole server, not only the MySQL Server through mysqldump, MySQL Enterprise Backup or any other solution. These backups can only be used to restore MySQL Server in another Azure Database for MySQL Server. It means that these backups cannot be exported to generate a new database on our on-premise server for instance. If we want to extract part of our database in order to export data, we can use mysqldump, MySQL Shell’s instance dump utility or the set of tools provided by MySQL Shell.

The backups provided by Azure can be used to make Point In Time Recovery of the Server with a granularity of 5 minutes since the system snapshots are done automatically every 5 minutes. As specified in the documentation, the backups are encrypted using AES 256-bit.

Backup and restore costs

As explained on Microsoft website

“Backup storage is the storage associated with automated backups of your server. Increasing your backup retention period increases the backup storage that is consumed by your server. There is no additional charge for backup storage for up to 100% of your total provisioned server storage. Additional consumption of backup storage will be charged in GB/month.” – https://azure.microsoft.com/en-us/pricing/details/mysql/flexible-server/

Increasing the retention of the backup, may have an impact on the pricing. We can have an idea of the global costs related to Azure Database for MySQL on the following URL: https://azure.microsoft.com/en-us/pricing/details/mysql/flexible-server/

Recover a database from Azure Backup Restore interface

In this first test, we will simply use the recovery functionalities provided by Azure Database for MySQL flexible server. We will simulate a user error by deleting a table and we will restore the entire server (as it’s not possible to simply recover a database or a table using Azure features).

1. Dropping a table by mistake

 MySQL  albatroz.mysql.database.azure.com:3306 ssl  SQL > SELECT CURRENT_TIMESTAMP ;
+---------------------+
| CURRENT_TIMESTAMP   |
+---------------------+
| 2022-08-18 20:48:44 |
+---------------------+
1 row in set (0.1002 sec)
 MySQL  albatroz.mysql.database.azure.com:3306 ssl  SQL > drop table sysbench.sbtest1;
Query OK, 0 rows affected (0.1884 sec)

2. Restoring the MySQL Server to a time before the mistake

We first have to go in the “backup/restore” menu of our Azure Database for MySQL flexible server and select the backupset that we want to restore. As we can see a backupset is done everyday. In our current context, we want to use the most recent backupset (Automated backup #5)

Once the backupset selected a screen appears showing the restore server options. It provides us with the possibility to make a Point In Time Restore (PITR) of our server by choosing between 3 options:

• Latest restore point (Now)
• Select a custom restore point
• Select fastest restore point (Restore using full backup)

In our case we will use the “Select a Custom Restore point” option as shown in the screenshot below. We will define the custom restore time just before the mistake and specify a name for the restored Server.

Once the restore requested, it took approximately 5 minutes to deploy the new server.

3. Let’s check if my table is back on the new deployed server

Finally we simply have to connect to the new restored server and check if the dropped table is back. Of course we can also export this table from this restored server and import this same table on the original server using mysqldump.

 MySQL  albatrozrestored.mysql.database.azure.com:3306 ssl  sysbench  SQL > show tables from sysbench like '%1';
+-------------------------+
| Tables_in_sysbench (%1) |
+-------------------------+
| sbtest1                 |
+-------------------------+
1 row in set (0.1048 sec)

4. Export/Import table from the restored Server

Now that the server is restored, we can export the table that was deleted by mistake using util.dumpTables() and import it to the albatroz server using util.loadDump(). The process is rather simple as you can see below:

Export from the recovered server (albatrozrestored)

MySQL  albatrozrestored.mysql.database.azure.com:3306 ssl  sysbench  JS > util.dumpTables("sysbench", [ "sbtest1"], "C:/Users/grs/Albatroz-Sysbench-sbtest1");
NOTE: Backup lock is not available to the account 'grs'@'%' and DDL changes will not be blocked. The dump may fail with an error if schema changes are made while dumping.
Acquiring global read lock
Global read lock acquired
Initializing - done
...
...
109% (15.29K rows / ~13.98K rows), 0.00 rows/s, 0.00 B/s uncompressed, 0.00 B/s compressed
Dump duration: 00:00:01s
Total duration: 00:00:06s
Schemas dumped: 1
Tables dumped: 1
Uncompressed data size: 2.93 MB
Compressed data size: 1.33 MB
Compression ratio: 2.2
Rows written: 15294
Bytes written: 1.33 MB
Average uncompressed throughput: 2.35 MB/s
Average compressed throughput: 1.07 MB/s

Import on Albatroz server

MySQL  albatroz.mysql.database.azure.com:3306 ssl  JS > util.loadDump("C:/Users/grs/Albatroz-Sysbench-sbtest1", {schema: "sysbench"});
Loading DDL and Data from 'C:/Users/grs/Albatroz-Sysbench-sbtest1' using 4 threads.
Opening dump...
Target is MySQL 8.0.28. Dump was produced from MySQL 8.0.28
Scanning metadata - done
Checking for pre-existing objects...
Executing common preamble SQL
Executing DDL - done
Executing view DDL - done
Starting data load
1 thds loading | 100% (2.93 MB / 2.93 MB), 1.94 MB/s, 0 / 1 tables done
Executing common postamble SQL
Recreating indexes - done
1 chunks (15.29K rows, 2.93 MB) for 1 tables in 1 schemas were loaded in 6 sec (avg throughput 1.94 MB/s)
0 warnings were reported during the load.

Recover a database from your own backups

As stated in the introduction, I will present in this chapter a complementary solution of backup/restore. Of course Azure does not prevent us from doing our own backups by connecting to the Azure Database for MySQL flexible server and using either mysqdump, MySQL Enterprise Backup or any other MySQL backup solution. I decided to use the opportunity of this blog to use the backup tools provided by MySQL Shell. Indeed MySQL Shell’s instance dump utility, such as util.dumpInstance(), util.dumpSchemas() or even util.dumpTables(), introduced in MySQL Shell 8.0.22, provide interesting functionalities. This export tools alone would deserve several blogs dedicated to them.

Before starting let’s illustrate what will be demonstrated in the next few lines:

1. The very first step consist in doing a dump of the MySQL Instance
2. Secondly, we will insert a row in a table named sbtest1
3. Thirdly, we will simulate a human error and drop a table
4. Then we will restore the database to the state after the backup
5. After having restored the backup, we will execute the binary to the state just before the human error
6. Finally we will check that the last insert I did is stored into the sbtest2

1. Dump of MySQL Instance using util.dumpInstance()

As explained above, the very first step consist in doing a dump of the entire instance. Without this backup we won’t be able to restore the database. We will use util.dumpInstance() as presented below:

MySQL  albatroz.mysql.database.azure.com:3306 ssl  JS > util.dumpInstance("C:/Users/grs/AlbatrozDump", {dryRun: false, showProgress: true, threads: 2})
NOTE: Backup lock is not available to the account 'grs'@'%' and DDL changes will not be blocked. The dump may fail with an error if schema changes are made while dumping.
Acquiring global read lock
Global read lock acquired
Initializing - done
2 out of 6 schemas will be dumped and within them 9 tables, 0 views.
4 out of 7 users will be dumped.
...
...
107% (137.28K rows / ~128.10K rows), 14.42K rows/s, 2.61 MB/s uncompressed, 1.20 MB/s compressed
Dump duration: 00:00:15s
Total duration: 00:00:21s
Schemas dumped: 2
Tables dumped: 9
Uncompressed data size: 26.26 MB
Compressed data size: 11.97 MB
Compression ratio: 2.2
Rows written: 137284
Bytes written: 11.97 MB
Average uncompressed throughput: 1.72 MB/s
Average compressed throughput: 784.25 KB/s

2. Inserting row in our table

Now we simulate some activity in the database by inserting a row into the table sbtest1.

MySQL  albatroz.mysql.database.azure.com:3306 ssl  sysbench  SQL > insert into sbtest1 values(999999999,1,1,"my row before drop table");
Query OK, 1 row affected (0.1134 sec)
MySQL  albatroz.mysql.database.azure.com:3306 ssl  sysbench  SQL > SELECT CURRENT_TIMESTAMP ;
+---------------------+
| CURRENT_TIMESTAMP   |
+---------------------+
| 2022-08-19 16:42:12 |
+---------------------+

3. Dropping a table by mistake

Thirdly, we simulate the human error by dropping the table sbtest1.

 MySQL  albatroz.mysql.database.azure.com:3306 ssl  sysbench  SQL > SELECT CURRENT_TIMESTAMP ;
+---------------------+
| CURRENT_TIMESTAMP   |
+---------------------+
| 2022-08-19 16:46:47 |
+---------------------+
1 row in set (0.1095 sec)
 MySQL  albatroz.mysql.database.azure.com:3306 ssl  sysbench  SQL > drop table sysbench.sbtest1;
Query OK, 0 rows affected (0.1823 sec)

4. Restoring the MySQL Server using the backup

Now, we have to restore the database using the last backup we have. We will use util.loadDump() in order to restore our table. To only recover the table we can simply use the option “includeTables“.

 MySQL  albatroz.mysql.database.azure.com:3306 ssl  sysbench  JS > util.loadDump("C:/Users/grs/AlbatrozDump", { includeTables: ["sysbench.sbtest1"],loadDdl:true, LoadData:true, threads: 2})
Loading DDL and Data from 'C:/Users/grs/AlbatrozDump' using 2 threads.
Opening dump...
Target is MySQL 8.0.28. Dump was produced from MySQL 8.0.28
Scanning metadata - done
Checking for pre-existing objects...
Executing common preamble SQL
Executing DDL - done
Executing view DDL - done
Starting data load
1 thds loading \ 100% (2.93 MB / 2.93 MB), 1.16 MB/s, 0 / 1 tables done
Executing common postamble SQL
Recreating indexes - done
1 chunks (15.29K rows, 2.93 MB) for 1 tables in 1 schemas were loaded in 9 sec (avg throughput 1.16 MB/s)
0 warnings were reported during the load.

If the restore worked properly, the table sbtest1 should be restored. We now have recovered the table sbtest1 without the queries that have been executed afterwards (before the drop table).

 MySQL  albatroz.mysql.database.azure.com:3306 ssl  sysbench  SQL > show tables;
+--------------------+
| Tables_in_sysbench |
+--------------------+
| sbtest1            |
| sbtest10           |
| sbtest2            |
| sbtest3            |
| sbtest4            |
| sbtest5            |
| sbtest6            |
| sbtest7            |
| sbtest8            |
| sbtest9            |
+--------------------+
10 rows in set (0.1169 sec)

MySQL  albatroz.mysql.database.azure.com:3306 ssl  sysbench  SQL > select * from sbtest1 where pad like 'my%';
Empty set (0.1284 sec)

5. Execution of the binary logs

Before executing the binary logs, we need to define what we call the “start-position” and the “end-position“. In order to find these two numbers, we have to find the log position after the backup and to find the log position of the “drop table”. The first (start-position) can be found into the metadata of the dump ( .json file). For the second we have to find the exact position using mysqlbinlog as demonstrated below (drop position at 14329648)

mysqlbinlog --verify-binlog-checksum --host=albatroz.mysql.database.azure.com --port=3306 --user=grs -p -                                                   -read-from-remote-server --verbose --start-datetime="2022-08-19 18:40:40" --stop-datetime="2022-08-19 18:50:47" mysql-bin.00                                                   0006 | grep -C 15 "DROP TABLE"

# at 14329648
#220819 18:47:12 server id 3691359094  end_log_pos 14329725 CRC32 0xa81066cf    Anonymous_GTID  last_committed=6652     sequence_number=6653r                                  br_only=no      original_committed_timestamp=1660927632152016   immediate_commit_timestamp=1660927632152016     transaction_length=217
# original_commit_timestamp=1660927632152016 (2022-08-19 18:47:12.152016 CEST)
# immediate_commit_timestamp=1660927632152016 (2022-08-19 18:47:12.152016 CEST)
/*!80001 SET @@session.original_commit_timestamp=1660927632152016*//*!*/;
/*!80014 SET @@session.original_server_version=80028*//*!*/;
/*!80014 SET @@session.immediate_server_version=80028*//*!*/;
SET @@SESSION.GTID_NEXT= 'ANONYMOUS'/*!*/;
# at 14329725
#220819 18:47:12 server id 3691359094  end_log_pos 14329865 CRC32 0x9f7865b0    Query   thread_id=480   exec_time=0     error_code=0    Xid =                                   412469
use `sysbench`/*!*/;
SET TIMESTAMP=1660927632/*!*/;
DROP TABLE `sbtest1` /* generated by server */
/*!*/;

Now that we have the start and end position of the binary log, we can apply the events in binary log file to the server. For my part, I prefer to go through an intermediate step consisting in creating a file containing all the events. This way I can look at what is inside and then just run my file. This allows me, for example, to see if I have made an error. As shown below I’m for instance able to show my “insert” statement:

osboxes@osboxes:~$  mysqlbinlog --verify-binlog-checksum --host=albatroz.mysql.database.azure.com --port=3306 --user=grs -p --read-from-remote-server --start-datetime="2022-08-19 16:40:40" --stop-datetime="2022-08-19 16:46:47" mysql-bin.000006 >/tmp/restore.sql
Enter password:
osboxes@osboxes:~$ vi /tmp/restore.sql
...
# at 14329545
#220819 18:41:18 server id 3691359094  end_log_pos 14329617 CRC32 0xd124e6a0    Write_rows: table id 913 flags: STMT_END_F

BINLOG '
Lr3/YhN2qwXcRQAAAMmm2gAAAJEDAAAAAAEACHN5c2JlbmNoAAdzYnRlc3QxAAQDA/7+BO7g/vAA
AQEAAgP8/wDqexN/
Lr3/Yh52qwXcSAAAABGn2gAAAJEDAAAAAAEAAgAE/wD/yZo7AQAAAAEAMRhteSByb3cgYmVmb3Jl
IGRyb3AgdGFibGWg5iTR
'/*!*/;
### INSERT INTO `sysbench`.`sbtest1`
### SET
###   @1=999999999
###   @2=1
###   @3='1'
###   @4='my row before drop table'
# at 14329617
...

Finally we can execute our restore script on the database.

osboxes@osboxes:~$ mysql --host=albatroz.mysql.database.azure.com --port=3306 --user=grs -p </tmp/restore.sql
Enter password:

6. Let’s check that our last insert has been executed

As we can see, the last record we inserted in the table is now present.

 MySQL  albatroz.mysql.database.azure.com:3306 ssl  sysbench  SQL > select * from sbtest1 where pad like 'my%';
+-----------+---+---+--------------------------+
| id        | k | c | pad                      |
+-----------+---+---+--------------------------+
| 999999999 | 1 | 1 | my row before drop table |
+-----------+---+---+--------------------------+

Conclusion

Azure Backup Restore interface provides an easy and interesting solution to backup and restore a MySQL Server going through the replication of the server. In addition in the tests I did, the deployment of the new server was rather fast. However, my server did not contain gigabytes of information. The 35 days of backup window could be seen as too small depending on your needs.

Beside of this solution I strongly encourage Database Administrators to keep on backing up their MySQL Database using other tools such as the ones provided by MySQL Shell or any other backup solution in order to ensure that no transactions are lost when restoring data. Such solutions could offer more flexibility in the backup and restore process and longer retention.

L’article MySQL Server on Microsoft Azure 3rd part (backup and recovery) est apparu en premier sur dbi Blog.

Introduction

This second blog follows the first blog about deploying MySQL Server on Microsoft Azure. In the first blog, we saw how easy it is to deploy a MySQL server in minutes on the Azure cloud and we connected on it through the MySQL Shell client.

This second blog is more focused on the performance of MySQL in the Azure Cloud. Although I didn’t do any tuning of MySQL parameters, we will see the influence of MySQL Server localization on the latency as well as changes in parameters such as CPU, memory or IOPS on performances using the SysBench tool.

SysBench stress test

Of course there are many tools available to do stress tests. On my side, I decided to use SysBench simply because I know this wonderful free tool and because it is widely deployed. Rather than making a description of SysBench, I simply take a part of its description available on sourceforge that we can find below:

sysbench is a scriptable multi-threaded benchmark tool based on LuaJIT. It is most frequently used for database benchmarks, but can also be used to create arbitrarily complex workloads that do not involve a database server – https://sourceforge.net/projects/sysbench.mirror/, 14.08.2022

Firstly I installed SysBench on my local Ubuntu client (osboxes). After the installation of SysBench but before starting the stress test, we first have to prepare the tables containing the records where the queries will be performed. In the following tests I created 10 tables with 1’000’000 rows in sysbench database (previously created).

osboxes@osboxes:/usr/bin$ sysbench --db-driver=mysql table-size=1000000 --mysql-host=albatroz.mysql.database.azure.com --mysql-port=3306 --mysql-db=sysbench --mysql-user=grs --tables=10 --mysql-password=MyPassword --test=/usr/share/sysbench/oltp_read_write.lua prepare

Once the tables populated, we can run the tests. For the tests, I decided to use oltp_read_write.lua which has a default ratio of read:write 95%:5%.

sysbench --db-driver=mysql  --num-threads=8 --mysql-user=grs --mysql-password=MyPassword --mysql-db=sysbench --events=0 --time=100  --test=/usr/share/sysbench/oltp_read_write.lua --mysql-host=albatroz.mysql.database.azure.com --mysql-port=3306 --tables=10 --db-ps-mode=disable --table-size=1000000 --report-interval=10 run

I varied the number of threads (num-threads parameter) from 8 to 256, limited the time to 100 seconds and changed two kinds of parameters through the Azure interface during my tests:

• The Server location (East US, vs West Switzerland)
• The Compute and Storage (Compute Tier and IOPS)

It’s important to note that my Client is also located in Western Switzerland to understand the impact of location. You can find below the four configurations I did:

1. Server located in US, with minimal compute and storage meaning 1vCore 2Gib Memory and 360 IOPS
2. Server located in US, with general purpose configuration meaning 2vCores, 8Gib Memory and 3200 IOPS
3. Server located in West Switzerland, with minimal compute and storage meaning 1vCore 2Gib Memory and 360 IOPS
4. Server located in West Switzerland, with general purpose configuration meaning 2vCores, 8Gib Memory and 3200 IOPS

Server details

As it’s not possible to change the location of a server afterwards, I created two MySQL servers in same version (version 8.0.28). One located in East US and the other located in West Switzerland. Note that we have two locations in Switzerland (North Switzerland and West Switzerland). As explained above, in order to make comparisons, I provided the same set of resources for tests 1 and 3 and test 2 and 4. Also as specified before, my client is located in Switzerland.

The basic configuration looks like the one below:

It is worth mentioning that to upgrade the server’s capacity (from Burstable to General Purpose with more IOPS), it only takes about ten minutes maximum. Meaning that when we end with the tests and we want to go to production, we only need few minutes to upgrade the server properties. We can find the description of the upgraded server below as well as the monthly price of this upgraded server.

Let’s run the tests

I executed the same tests with only a variation in the number of thread (from 8 to 256) on each four configurations. We can find an extract of the execution as well as the output below

osboxes@osboxes:/usr/bin$ sysbench --db-driver=mysql  --num-threads=8 --mysql-user=grs --mysql-password=MyPassword --mysql-db=sysbench --events=0 --time=100  --test=/usr/share/sysbench/oltp_read_write.lua --mysql-host=albatroz.mysql.database.azure.com --mysql-port=3306 --tables=10 --db-ps-mode=disable --table-size=1000000 --report-interval=10 run
WARNING: the --test option is deprecated. You can pass a script name or path on the command line without any options.
WARNING: --num-threads is deprecated, use --threads instead
sysbench 1.0.20 (using system LuaJIT 2.1.0-beta3)

Running the test with following options:
Number of threads: 8
Report intermediate results every 10 second(s)
Initializing random number generator from current time


Initializing worker threads...

Threads started!

[ 10s ] thds: 8 tps: 3.20 qps: 77.58 (r/w/o: 55.98/4.40/17.20) lat (ms,95%): 2082.91 err/s: 0.00 reconn/s: 0.00
[ 20s ] thds: 8 tps: 4.00 qps: 72.91 (r/w/o: 50.51/5.00/17.40) lat (ms,95%): 2585.31 err/s: 0.00 reconn/s: 0.00
[ 30s ] thds: 8 tps: 4.00 qps: 77.80 (r/w/o: 53.80/5.30/18.70) lat (ms,95%): 2120.76 err/s: 0.00 reconn/s: 0.00
[ 40s ] thds: 8 tps: 3.20 qps: 72.80 (r/w/o: 52.50/4.50/15.80) lat (ms,95%): 2632.28 err/s: 0.00 reconn/s: 0.00
[ 50s ] thds: 8 tps: 4.00 qps: 78.40 (r/w/o: 55.40/5.10/17.90) lat (ms,95%): 2082.91 err/s: 0.00 reconn/s: 0.00
[ 60s ] thds: 8 tps: 4.00 qps: 73.60 (r/w/o: 49.70/5.40/18.50) lat (ms,95%): 2632.28 err/s: 0.00 reconn/s: 0.00
[ 70s ] thds: 8 tps: 4.00 qps: 77.60 (r/w/o: 53.60/5.00/19.00) lat (ms,95%): 2082.91 err/s: 0.00 reconn/s: 0.00
[ 80s ] thds: 8 tps: 3.20 qps: 73.60 (r/w/o: 54.10/4.70/14.80) lat (ms,95%): 2405.65 err/s: 0.00 reconn/s: 0.00
[ 90s ] thds: 8 tps: 4.00 qps: 77.60 (r/w/o: 53.90/4.80/18.90) lat (ms,95%): 2082.91 err/s: 0.00 reconn/s: 0.00
[ 100s ] thds: 8 tps: 4.00 qps: 73.60 (r/w/o: 49.60/4.80/19.20) lat (ms,95%): 2493.86 err/s: 0.00 reconn/s: 0.00
SQL statistics:
    queries performed:
        read:                            5376
        write:                           501
        other:                           1803
        total:                           7680
    transactions:                        384    (3.78 per sec.)
    queries:                             7680   (75.60 per sec.)
    ignored errors:                      0      (0.00 per sec.)
    reconnects:                          0      (0.00 per sec.)

General statistics:
    total time:                          101.5914s
    total number of events:              384

Latency (ms):
         min:                                 2000.71
         avg:                                 2115.55
         max:                                 2644.11
         95th percentile:                     2585.31
         sum:                               812371.47

Threads fairness:
    events (avg/stddev):           48.0000/0.00
    execution time (avg/stddev):   101.5464/0.07

We can see in this output, that I asked to sysbench to provide me with a feedback every 10 seconds. We can also observe that the ratio between the queries and transaction is respected (95%:5%), finally it can noted that the latency is important in our case (Server in US and client in Switzerland). The average latency in the above example is 2’115.55ms. If we are interested in the latency distribution we can use the option –histogram.

Tests summary

We can find below the summary of the tests I did with the described configurations.

The first thing we can see after having a look on the above table is that whereas we faced a “too many connections” errors with minimal configurations, the tests worked properly with the General purpose configuration.

The second thing that we can observe is that the tests performed better with the server located in West Switzerland. One of the reasons is most probably linked with the latency. Whereas we had an average of 2’100ms of latency for server located in East US, we have an average of 350ms of latency for server located in West Switzerland.

Finally in our configuration, the increase of compute and storage doesn’t show any improvement in terms of transaction per second or queries per second.

Latency between the application and the MySQL Server

The latency is an important parameter to take into consideration. As explained on Microsoft webpage named “Best practices for optimal performance of your Azure Database for MySQL server” to improve the performance of an application, we have to take into consideration the proximity between the MySQL Server and the application.

To improve the performance and reliability of an application in a cost optimized deployment, it’s highly recommended that the web application service and the Azure Database for MySQL resource reside in the same region and availability zone. – Microsoft, 16.08.2022

This page also gives some tips in order to optimize your MySQL performances on Azure. To check the latency, you do not need to use SysBench, a simple query such as the one executed in the below examples presents the latency difference between the execution on a localhost, an Azure server located in Switzerland and an Azure server located in US.

Latency on a Localhost

mysql> select 1;
+---+
| 1 |
+---+
| 1 |
+---+
1 row in set (0.00 sec)

Latency on an Azure Server located in West Switzerland

 MySQL  albatrozswitzerland.mysql.database.azure.com:3306 ssl SQL > select 1;
+---+
| 1 |
+---+
| 1 |
+---+
1 row in set (0.0436 sec)

Latency on an Azure Server located in East US

 MySQL  albatroz.mysql.database.azure.com:3306 ssl SQL > select 1;
+---+
| 1 |
+---+
| 1 |
+---+
1 row in set (0.1690 sec)

Considering only the latency, one could think that the application must be hosted on an Azure server in Switzerland (at least in my case). Unfortunately we will discover that not all services of the Microsoft Azure Cloud are available in Switzerland. For instance if we want to deploy an Ubuntu Server, we will see that only the following location are available:

Performance monitoring

Microsoft Azure provides us with an interface showing the following default graphs (cf screenshot below:

• CPU and Memory
• IO Percent
• DB Connections
• Queries

But you can add other information regarding storage, Host network in/out, replication lag, aborted connection, aso…

Conclusion

This short blog regarding the performance of MySQL in the Azure Cloud provides the following conclusions:

The ability to upgrade a MySQL server by adding CPU, disk and memory capacity in a few minutes is really interesting. For instance when you want to switch your test project to production, you simply need to change the servers properties and 10 minutes later, your server is adapted. Additionally, we can always reduce the CPU & Memory capacity as well as the IOPS, only the storage cannot be reduced.

At first glance, it also seems that it is very important to use a MySQL server close to the client for performance reasons (in order to avoid latency issues). Unfortunately depending on the application and business requirements it could be difficult to have the MySQL Server and the Application Server in the same country.

Finally adding CPU, memory and increasing the number of IOPS did not show any improvement in the number of transactions or queries per second in this specific configuration (of course it’s strongly depending on the use case). However, this does not mean that by taking the time to configure the MySQL server correctly, one could not benefit from these additional capacities. Regarding the tuning of MySQL variables, it’s interesting to see that some variables are updated automatically such as the innodb_buffer_pool_size when we change server resources.

L’article MySQL Server on Microsoft Azure 2nd part (Performance tests) est apparu en premier sur dbi Blog.

Introduction

Did you know that you can run MySQL on Microsoft Azure for free for 30 days with a $200 credit? In this first blog I’ll show how to create a MySQL Server and provide few information related to this service. In futures blogs I’ll present insights regarding MySQL performances and backup/recovery.

Let’s start by registering ourselves

The registration process lasts approximately 5 to 10 minutes. We simply have to enter our contacts details as well as our credit card details. Don’t worry, no fees are automatically charged. Once your 200$ credit limit used, Microsoft will ask us if we want to keep on with the payment of the additional fees.  

In order to register ourselves on Microsoft Azure, we can simply go on the following URL: https://azure.microsoft.com/en-us/services/mysql/ and click on “Try Azure Database for MySQL free” and “Start free”. Then we enter our contact details such as country, name, surname, phone number, e-mail address, address, aso… Once finished, we will see a screen looking like the one below. We simply have to select the time zone and if we want to join a Q&A session.

Our first MySQL Server on Azure

Once the registration process ended, it’s time to create our first MySQL Server on Azure. Log in into Microsoft Azure and enter “mysql” in the research field as shown below and select “Azure Database for MySQL servers”.  

Once “Azure Database for MySQL servers” selected, we simply have to click on “Create Azure Database for MySQL Server” on the next screen. We now have to select the deployment option. As shown below, we have the choice between the two following options:

• Flexible Server (Recommended)
• Single Server

In the context of this blog, I chose the first option (Flexible server).

Before deploying our first MySQL Server we have to follow a straightforward process which requires 4 steps described below. As we will see, depending on the workload type retained (which will affect the CPU, memory and possible IOPS), the storage and IOPS, the monthly fees will vary between USD 18.98/month and USD >10’000/month.

What is really nice is to be able to see in real time the influence that each parameter has on the price estimate on the right hand side as we can see in the screenshot below.

1. Firstly, we have to enter basic information regarding our MySQL Server such as :
   • Subscription details,
   • Server name
   • Region where you want to deploy your server
   • MySQL Version (version 5.x or 8.x)
   • Workload type (Small or Medium Size database, Tier 1 Business Critical Workloads, for development of hobby projects)
   • Compute+Storage: Compute options – (Burstable 1-20 vCores, General purpose 2-64 vCores or Business Critical 2-96 vCores). Storage options – from 20GB up to 16384GB (no scale down possibility) and from 360 to 48000IOPS
   • Availability Zone (optional)
   • Enable High-Availability (optional)
   • Administrator account
     
     
2. Secondly, we have to fill in networking information regarding our MySQL Server such as:
   • Network Connectivity (connect to your server through a public IP address or a Private Access)
   • Firewall rules (you can automatically select your current IP address or allow any remote connection)
     
     
3. Thirdly, we can enter tags. Tags are name/value pairs that enable you to categorize and view consolidated billing by applying the same tag to multiple resources and resource groups.
   
   
4. We can finally review and create our server. Please note that we can also download a template for automation purpose.

In the compute option, you can select few options as presented below:

Once all the the fields completed we can create our server. It’s interesting to see that we also have the possibility to generate a template for automation purpose.

On the summary page, you can see that you have access to the Terms of Use and Privacy Policy. In the Terms of Use we can find a link to the SLA Conditions where we can find the guaranteed service time as well as the credit in case of non compliance with the SLA conditions.

The deployment and start-up of the server takes approximately 6 minutes.

We have a server, let’s use it now!

In order to use our brand new MySQL Server, I decided to use MySQL Shell. MySQL Shell is a MySQL Client that provides scripting capabilities in JavaScript and Python. You can download MySQL Shell on this link and find MySQL Documentation on this link.

When we run MySQL Shell on Windows, a cmd window opens with a command prompt as shown below. To connect to our MySQL Server, we can use the “connect” keyword. and then switch in sql mode with the “\sql” command.

MySQL  JS > \connect --mysql albatroz.mysql.database.azure.com
Creating a Classic session to 'grs@albatroz.mysql.database.azure.com'
Please provide the password for 'grs@albatroz.mysql.database.azure.com': *************
Save password for 'grs@albatroz.mysql.database.azure.com'? [Y]es/[N]o/Ne[v]er (default No): yes
Fetching schema names for autocompletion... Press ^C to stop.
Your MySQL connection id is 20
Server version: 8.0.28 Source distribution
No default schema selected; type \use <schema> to set one.
 MySQL  albatroz.mysql.database.azure.com:3306 ssl  JS > \sql
Switching to SQL mode... Commands end with ;

Using show variables like ‘%version%’; command provides us few information regarding the MySQL version as well as the Operating System in use. Version 8.0.28 is not the latest version but it is not old either since it dates from December 2021 and at the time of writing this blog, the latest version is 8.0.30.

MySQL  albatroz.mysql.database.azure.com:3306 ssl  SQL > show variables like '%version%';
+--------------------------+---------------------+
| Variable_name            | Value               |
+--------------------------+---------------------+
| admin_tls_version        | TLSv1.2             |
| immediate_server_version | 999999              |
| innodb_version           | 8.0.28              |
| original_server_version  | 999999              |
| protocol_version         | 10                  |
| replica_type_conversions |                     |
| slave_type_conversions   |                     |
| tls_version              | TLSv1.2             |
| version                  | 8.0.28              |
| version_comment          | Source distribution |
| version_compile_machine  | x86_64              |
| version_compile_os       | Linux               |
| version_compile_zlib     | 1.2.11              |
+--------------------------+---------------------+

After few tests, we can see some performance graphs (CPU and Memory, IO Percent, DB Connections and Queries) through the Azure interface.

Conclusion

This first blog does not pretend to go into detail about the possibilities offered by Azure for the deployment of a MySQL server. It simply shows that deploying a MySQL server on the Azure Cloud is really simple. In addition, the $200 offered by Microsoft on the first 30 days of use allows you to have a first overview of the possibilities but also the costs related to a deployment in the Azure Cloud. The interface allows anyone to easily deploy a MySQL database in minutes.

The next blog will discuss MySQL performance on the Azure Cloud through stress tests performed with SysBench.

L’article MySQL Server on Microsoft Azure 1st part (deployment) est apparu en premier sur dbi Blog.

Are you experiencing performance problems with your Zyxel ZyWALL USG firewall? You will find in this blog an example of a real case I faced and how I solved it.

Introduction

From one day to the next, network performance deteriorated. Users first reported sporadic access problems and then these problems became permanent. The performance problems resulted in very slow access to a remote system. As shown below, the pings performed sometimes went up to more than a hundred milliseconds,

Pinging 8.8.8.8 with 32 bytes of data
Reply from 8.8.8.8: Bytes=32 time=12 ms TTL=116
Reply from 8.8.8.8: Bytes=32 time=63 ms TTL=116
Reply from 8.8.8.8: Bytes=32 time=12 ms TTL=116
Reply from 8.8.8.8: Bytes=32 time=562 ms TTL=116
Reply from 8.8.8.8: Bytes=32 time=519 ms TTL=116
Reply from 8.8.8.8: Bytes=32 time=211 ms TTL=116
Reply from 8.8.8.8: Bytes=32 time=31 ms TTL=116
Reply from 8.8.8.8: Bytes=32 time=617 ms TTL=116
Reply from 8.8.8.8: Bytes=32 time=100 ms TTL=116
Reply from 8.8.8.8: Bytes=32 time=19 ms TTL=116

Thanks to UDITIS, we were able to quickly isolate the problem and understood that it came from the firewall. In this case it was a ZyWALL USG 50 firewall.

Logging into the ZyWALL USG GUI, we quickly saw that the processor resources were being used at over 90%. This had an impact on packet processing.

Problem analysis

Unfortunately the ZyWALL graphical interface does not give more details about the use of resources and does not allow to see what the processor or memory resources are used for. To analyze the problem, it is therefore necessary to connect to the Zywall via SSH.

As explained on the Zyxel support webpage, we can use the command line “debug system show cpu status” to see the CPU usage details. For instance, if CPU time would have been spent on “softirq” it would have mean that the CPU was occupied with traffic load. In our case the CPU is occupied with a system item (currentrly 51%) and by user (currently 47%).

login as: admin
Using keyboard-interactive authentication.
Password:
Bad terminal type: "xterm". Will assume vt100.

Router> debug system show cpu status
CPU utilization: 99 % (system: 51 %, user: 47 %, irq: 0 %, softirq 1 %)
CPU utilization (1 minute): 98 % (system: 48 %, user: 43 %, irq: 4 %, softirq 3 %)
CPU utilization (5 minute): 98 % (system: 47 %, user: 44 %, irq: 4 %, softirq 3 %)

To understand why the CPU is used by system item, we can use the command “debug system ps“. This command will help to see the different processes and the resources usage as presented below (extract).

Router> debug system ps
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.0    208   100 ?        Ss   Aug05   0:02 ini
root         2  0.0  0.0      0     0 ?        S<   Aug05   1:01 [kthreadd]
root       816  0.3  0.5   8464  1240 ?        Ss   Aug05   5:20 /usr/sbin/zylogd
root       820  0.0  0.1   1104   444 ?        Ss   Aug05   0:05 /usr/sbin/syslog-ng -f /var/zyxel/syslog-ng/syslog-ng.conf
root      1527  0.0  0.1   3760   404 ?        S    Aug05   0:00 /usr/sbin/pcap_monitor
root      1533  0.0  0.1   8920   340 ?        Ss   Aug05   0:00 /usr/sbin/vsd
root      1536  0.0  0.1   2588   276 ?        S<   Aug05   0:00 /bin/zyshd_wd
root      1540  0.3  4.8  54304 11708 ?        S    Aug05   5:00 /bin/zyshd
root      1696  0.0  0.2   7068   664 ?        Ss   Aug05   0:00 /usr/sbin/xinetd -filelog /tmp/xinetd.log -stayalive -reuse -pidfile /var/run/xinetd.pid
root      1807  0.0  1.0  50108  2456 ?        Ss   Aug05   0:00 /usr/sbin/radiusd -d /var/zyxel/raddb
root      2297  0.0  0.1   2704   408 ?        S<   Aug05   0:00 /sbin/resd
root      2303  0.0  0.8  35436  1944 ?        Ss   Aug05   0:04 /usr/sbin/contfltd
root      2382  0.0  0.0   2572   196 ?        Ss   Aug05   0:00 /sbin/lavd
root      2383  0.0  1.1  15836  2768 ?        S<   Aug05   0:00 /sbin/decomp_server
root      2397  0.0  0.2  16212   632 ?        Ss   Aug05   0:00 /usr/sbin/fauthd
root      2412  0.0  0.1   2772   320 ?        S<   Aug05   0:00 /sbin/wdtd
...
...
nobody   12843  0.0  1.5  41860  3804 ?        S    Aug05   0:00 /usr/local/apache/bin/httpd -f /usr/local/zyxel-gui/httpd.conf -k start -DSSL
nobody   12851  0.0  1.5  41860  3688 ?        S    Aug05   0:00 /usr/local/apache/bin/httpd -f /usr/local/zyxel-gui/httpd.conf -k start -DSSL
nobody   12852  0.0  1.6  44004  4108 ?        S    Aug05   0:00 /usr/local/apache/bin/httpd -f /usr/local/zyxel-gui/httpd.conf -k start -DSSL
root     20680 91.9  1.2  26976  3076 ?        R    10:18   4:31 /usr/local/sbin/snmpd udp:161,udp6:161 -c /var/zyxel/snmpd.conf -p /var/run/snmpd.pid
...
...

As you can see, the SNMP process is using 91.9% of CPU usage. SNMP monitoring can be used to collect information from your USG. If you do not need it you can simply deactivate it through the graphical interface. In addition, it looks to be a good practice on the USG when not using this feature. In order to deactivate the SNMP functionality just go in Configuration section and then in System > SNMP as shown below:

Once SNMP deactivated, Processor usages are freed up.

Router> debug system show cpu status
CPU utilization: 12 % (system: 5 %, user: 2 %, irq: 3 %, softirq 2 %)
CPU utilization (1 minute): 78 % (system: 39 %, user: 33 %, irq: 3 %, softirq 3 %)
CPU utilization (5 minute): 92 % (system: 45 %, user: 41 %, irq: 3 %, softirq 3 %)

The pings are now much more stable and there are no more spikes as we saw at the beginning.

Pinging 8.8.8.8 with 32 bytes of data
Reply from 8.8.8.8: Bytes=32 time=12 ms TTL=116
Reply from 8.8.8.8: Bytes=32 time=11 ms TTL=116
Reply from 8.8.8.8: Bytes=32 time=12 ms TTL=116
Reply from 8.8.8.8: Bytes=32 time=11 ms TTL=116
Reply from 8.8.8.8: Bytes=32 time=12 ms TTL=116
...
...
Reply from 8.8.8.8: Bytes=32 time=12 ms TTL=116
Reply from 8.8.8.8: Bytes=32 time=12 ms TTL=116
Reply from 8.8.8.8: Bytes=32 time=12 ms TTL=116
Reply from 8.8.8.8: Bytes=32 time=12 ms TTL=116
Reply from 8.8.8.8: Bytes=32 time=12 ms TTL=116

Conclusion

In the event of a performance problem with a ZyWALL USG, the command line tool provided by ZyXEL allows you to understand what resources are being monopolized by. Once you have identified what was consuming the resources, in our case the SNMP process, you can either stop it or go further in the debugging process.

L’article Performance problems on a ZyWALL USG est apparu en premier sur dbi Blog.

A customer of mine asked me a licensing question regarding the usage or Oracle Java (OJVM) in an Oracle database.  Basically this customer have an Oracle Database Standard Edition License and would like to use Oracle JVM in order to load Java classes in his PL/SQL code executed by this database. The customer would like to ensure that his Oracle Database Standard Edition gives him the right to execute Java Classes in his PL/SQL code.

Sources of information

The question, even if looking obvious for some people, wasn’t so obvious for me.  Fact is that I wasn’t really able to find the information on Oracle documentation nor in a note on My Oracle Support. I started to ask to some of our internal Specialist such as Pascal Brand and Franck Pachot in order to get some piece of answer. I use this opportunity to thanks both of them for their feedbacks ;-).

Both provided me some valuable inputs:

Their understanding was that the Java License is included in Oracle Database Standard Edition according to the following documentation:

In addition they provided me some MOS (My Oracle Support) notes were it was more clearly explained such as the note 1557737.1. as well as some link to the Java FAQ where the answer was even more explicit.

“If you are an Oracle Customer with a supported Oracle product which requires Java SE, you continue to have access to Oracle Java updates, as required by your Oracle product, for the use of supported Oracle products, at no additional cost.”

Finally in order to be sure of my answer to the customer I wrote the following Service Request:

Q:
=======
A question regarding Oracle Database Standard Edition and Java.

We would like to use Oracle JVM in order to load Java classes in our PL/SQL code executed by a standard edition database. We would like to ensure that the Oracle Database Standard Edition gives us the right to execute Java Classes in PL/SQL code.

Our understanding is that the Java License is included in Oracle Database Standard Edition according to the following documentation:
https://docs.oracle.com/en/database/oracle/oracle-database/19/dblic/Licensing-Information.html#GUID-AB354617-6614-487E-A022-7FC9A5A08472

As far as we do understand we will however have to use the Java version provided by the Oracle Database located in the ORACLE_HOME. According to https://blogs.oracle.com/java-platform-group/oracle-java-se-releases-faq :
“If you are an Oracle Customer with a supported Oracle product which requires Java SE, you continue to have access to Oracle Java updates, as required by your Oracle product, for the use of supported Oracle products, at no additional cost.”

May you please confirm that our understanding is correct, meaning that with Oracle Database Standard Edition, the Oracle Java is included and therefore we can load java classes in PL/SQL code without additional license ?

The answer from the support was crystal clear:

A:
======
Yes, your understanding correct. Here in Oracle there are different Java products.
If you are going to use JAVAVM (Embedded JVM in Database) then DB license is enough. The embedded Database JVM is used to develop and load java source/ classes in database under custom schema.
These java stored procedures are wrapped with pl/sql and invoked by user.
ref: https://docs.oracle.com/en/database/oracle/oracle-database/19/jjdev/Java-application-strategy.html#GUID-492B0C81-24A3-4551-A151-BC0DCE23C802

Thank you.

Conclusion

Following these inputs I’ve been able to inform the customer that even with the Standard Database Edition he is fully allowed to load Java Classes in his PL/SQL Code. If you have doubts regarding usage of Java with Oracle products you will find some interesting information in the following note 1557737.1. I hope this small post will help some of you.

L’article Oracle Java Virtual Machine included in Oracle Database license ? est apparu en premier sur dbi Blog.

Computing environments became more and more agile over these last years. Companies need to provide solutions helping people to quickly set up new resources, starting and stopping them, scaling them according to the need and finally, removing them. In such environments, it could be tricky to follow license compliance when resources are changing on hourly basis.

Having a look on AWS services, I saw that AWS provides a license managing tool named “AWS License Manager”. I took few minutes in order to:

• Understand which resources this service is able to monitor
• How it works
• Test it with an on-premise Linux server executing an oracle database

License Manager Service

The first step in order to use License Manager is to select it in the list of AWS Services.

After having clicked on AWS License Manager, the AWS License Manager window will appear.

Now, we simply have to create a license configuration with required license terms according to the software vendor. You can setup different kind of metrics such as

• vPCUs
• Cores
• Sockets
• Instances

License Manager also provides the possibility to enforce license limit, meaning that it prevents license usage after available licenses are exhausted.

In a context of on-premise License monitoring, it is important to notice that sockets and cores license’s type are not accepted. Therefore, in this example I used vCPUs.

AWS System Manager

Once the license configuration created, it’s now mandatory to use another AWS Service, AWS System Manager. This service allows you to view and control your infrastructure on AWS. AWS System Manager not only allows you to view and control your Amazon EC2 Instance but also on-premises servers, virtual machines (including VMs in other cloud environments). Some System Manager capabilities are not free, however in the context of this example everything is free.

AWS System Manager Agent (SSM Agent)

In order to benefit from AWS System Manager we need to install AWS Systems Manager Agent (SSM Agent) on our on-premised host. Indeed, SSM Agent is an Amazon software that can be installed and configured on an Amazon EC2 instance, an on-premises server, or a virtual machine (VM) and provides a solution to update, manage, and configure resources. SSM Agent is installed, by default on instances created from Windows Server 2016 and Windows Server 2019, Amazon Linux, Ubuntu Server Images AMIs. However, if you are running an on-premise server you need to install it. The process is really straightforward as presented below.

[root@vmrefdba01 ~]# mkdir /tmp/ssm
[root@vmrefdba01 ~]# curl https://s3.amazonaws.com/ec2-downloads-windows/SSMAgent/latest/linux_amd64/amazon-ssm-agent.rpm -o /tmp/ssm/amazon-ssm-agent.rpm
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 18.9M  100 18.9M    0     0  3325k      0  0:00:05  0:00:05 --:--:-- 4368k
[root@vmrefdba01 ~]# sudo yum install -y /tmp/ssm/amazon-ssm-agent.rpm
Loaded plugins: refresh-packagekit, ulninfo
Setting up Install Process
Examining /tmp/ssm/amazon-ssm-agent.rpm: amazon-ssm-agent-2.3.707.0-1.x86_64
Marking /tmp/ssm/amazon-ssm-agent.rpm to be installed
public_ol6_UEK_latest                                    | 2.5 kB     00:00
public_ol6_UEK_latest/primary_db                         |  64 MB     00:07
public_ol6_latest                                        | 2.7 kB     00:00
public_ol6_latest/primary_db                             |  18 MB     00:07
Resolving Dependencies
--> Running transaction check
---> Package amazon-ssm-agent.x86_64 0:2.3.707.0-1 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

======================================================================================================================================
 Package                            Arch                     Version                        Repository                           Size
======================================================================================================================================
Installing:
 amazon-ssm-agent                   x86_64                   2.3.707.0-1                    /amazon-ssm-agent                    61 M

Transaction Summary
======================================================================================================================================
Install       1 Package(s)

Total size: 61 M
Installed size: 61 M
Downloading Packages:
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
  Installing : amazon-ssm-agent-2.3.707.0-1.x86_64                                                                                1/1
amazon-ssm-agent start/running, process 3896
  Verifying  : amazon-ssm-agent-2.3.707.0-1.x86_64                                                                                1/1

Installed:
  amazon-ssm-agent.x86_64 0:2.3.707.0-1

Complete!

Creating an activation

Once the agent installed, we have to create a new “Activation” in the AWS System Manager Service by clicking on “Create activation“. At the end of the creation you will get an Activation Code and an Activation ID (in the green field below). You have to keep this information for the agent configuration.

Agent Configuration

In order to register your on-premise instance on AWS, you simply have to execute the following command with the activation code and activation id provided by AWS System Manager

sudo amazon-ssm-agent -register -code "<cf Activation Code>" -id "<cf Activation ID>" -region "us-east-2"

2019-09-19 13:53:05 INFO Successfully registered the instance with AWS SSM using Managed instance-id: mi-0756a9f0dc25be3cd

Once registered the Managed Instance should appear as presented below in AWS Systems Manager

The Platform type is detected as well as the Kernel version, IP address and computer name. AWS Systems Manager provides also a package inventory and many other kinds of inventory such as Network inventory, Files inventory, aso…

Association between License Configuration and Resource ID

We now have to make the link between the Managed Instance (resource) and the license configuration. The goal of course is to define which license configuration will be applied to which resource. In order to proceed, we have to go into the AWS License Manager, and select “Search Inventory” into the menu. Then we simply have to select the resource and then click on “Associate license Configuration”.

The following window will appear, allowing you to define to which license configuration matches which resource:

Having a look in the AWS License Manager Dashboard, you can see that 1 out of 1 license is consumed since I dedicated one vCPU to my virtual machine and I provided 1vCPU  license to this instance.

Core Messages

• AWS License Manager offers more functionalities for EC2 Instances than for on-premise servers.
• AWS License Manager offers functionalities in order to monitor socket, vCPU, Cores and Instances.
• AWS License Manager definitively helps to manage licenses but doesn’t fit with all requirements and license model.
• AWS Systems Manager is a powerful tool providing several functionalities.

Strenghts

• AWS License Manager is free.
• AWS License Manager offers possibilities to monitor on-premise resources.
• AWS License Manager provides solution in order to prevent instance to run if license compliance doesn’t fit.
• AWS License Manager and AWS System Manager are straightforward to install and configure.
• AWS License Manager and AWS System Manager offers a good documentation.
• AWS System Manager offers many free functionalities (Patch Manager, Session Manager, Insights Dashboard, aso…).
• AWS System Manager offers many  functionalities and is the basis of several other AWS tools such as AWS Config which allows to monitor instance’s compliance.

Weaknesses

• AWS License Manager is not able by default to monitor options usage such as the ones of an Oracle database (Partitioning, Active Data Guard, aso…).
• AWS License Manager is not able to calculate Oracle Processors, meaning taking into consideration core factors.
• AWS System Manager is not able to monitor socket or cores on on-premise resources, only vCPUs.

L’article Managing Licenses with AWS License Manager est apparu en premier sur dbi Blog.