.
Here is how to quickly install the CLI (Command Line Interface) for the following public clouds: Amazon, Google, Microsoft and Oracle. On Linux, I’m using wget but you can use curl. I’ll install all cloud command line interfaces into a $HOME/cloud directory and add an alias into $HOME/.bashrc if not already existing

Amazon – AWS

AWS – install

( mkdir -p ~/cloud/aws-cli ; cd /var/tmp && wget -qc https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip && unzip -qo awscli-exe-linux-x86_64.zip && ./aws/install --update --install-dir ~/cloud/aws-cli --bin-dir ~/cloud/aws-cli && rm -rf ./awscli-exe-linux-x86_64.zip ./aws )
alias aws || echo 'alias aws=~/cloud/aws-cli/aws' >> ~/.bashrc && . ~/.bashrc
aws --version

AWS – update

As I used the –update flag in the install command line, and the url goes to the latest version, I just run the same to update the CLI

AWS – configure

• You need to get the “AWS Access Key ID” and “Secret access key” define for the user (go to https://console.aws.amazon.com/iam/home#/users, select the user, Security Credentials and Create Access Key). You cannot have many of them (I think only two for the time to change it in all CLI) so if you have one already created, just use the same (which means you have access to a previous CLI config).
• You can define the default region (which you find in the console menu, when you are in a regional service – go to https://console.aws.amazon.com/cloudshell/home and it will appear in the url)
• I define text as default output because others sometimes lie to me (see https://dev.to/aws-heroes/dynamodb-scan-and-why-128-5-rcu-2k25)

You find all info in one screen as you can see in the screenshot

$ aws configure

AWS Access Key ID [None]: AKIA3VX74TJVG6MJGSO7 
AWS Secret Access Key [None]: 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
Default region name [None]: eu-west-1
Default output format [None]: text

[opc@cern-exa18-ecktj1 ~]$ cat ~/.aws/config

[default]
region = eu-west-1
output = text
[opc@cern-exa18-ecktj1 ~]$ cat ~/.aws/credentials

[default]
aws_access_key_id = AKIA3VX74TJVG6MJGSO7 
aws_secret_access_key = 9AWS6denUHQxRGZlobSNmhXGzQE+1XKrsObgTJlZ

This is an example where you can set that the information is written into a ~/.aws directory, associated with a [default] profile. You can use a different profile with –profile or set all values with arguments and environment variables like:

$ AWS_ACCESS_KEY_ID=AKIA3VX74TJVG6MJGSO7 AWS_SECRET_ACCESS_KEY=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 AWS_DEFAULT_REGION=eu-west-1 aws dynamodb list-tables --region eu-west-1 --output json

{
    "TableNames": [
        "SkiLifts"
    ]
}

Of course, with those credentials you can access to my account, you must keep them safe and secret. And use the least priviledged user. I’ll remove them from my IAM account before publishing this post.

AWS – uninstall

To uninstall I remove the cli and the configuration:

rm -rf ~/cloud/aws-cli/aws ~/.aws

Google – GCP

GCLOUD – install

Here, unfortunately, the URL mentions the version and there’s no URL to get the latest one.

( mkdir -p ~/cloud/gcp-cli ; cd /var/tmp && wget -qc https://dl.google.com/dl/cloudsdk/channels/rapid/downloads/google-cloud-sdk-337.0.0-linux-x86_64.tar.gz && tar -zxf $(basename $_) -C ~/cloud/gcp-cli && ~/cloud/gcp-cli/google-cloud-sdk/install.sh --usage-reporting false --screen-reader false --command-completion false --path-update true --rc-path ~/cloud/.bashrc --override-components && rm -rf google-cloud-sdk-linux-x86_64.tar.gz && gcloud config set disable_usage_reporting false)
alias gcloud || echo 'alias gcloud=~/cloud/gcp-cli/google-cloud-sdk/bin/gcloud' >> ~/.bashrc && . ~/.bashrc
alias gsutil || echo 'alias gsutil=~/cloud/gcp-cli/google-cloud-sdk/bin/gsutil' >> ~/.bashrc && . ~/.bashrc
alias bq    || echo 'alias      bq=~/cloud/gcp-cli/google-cloud-sdk/bin/bq'     >> ~/.bashrc && . ~/.bashrc
gcloud --version

I choose to disable usage reporting here. And update a .bashrc in my ~/cloud directory rather than the default one

GCLOUD – update

As the url may get a past version, better to upgrade

$ yes | gcloud components update
Beginning update. This process may take several minutes.

This updates all components

GCLOUD – config

$ gcloud init

The authentication gives you an URL where you allow and get a verification code to paste. You can optionally set the default region and zone. Those are visible in ~/.config/gcloud

$ cat ~/.config/gcloud/configurations/config_default

[core]
disable_usage_reporting = false
account = myaccount@gmail.com
project = disco-abacus-424242

[compute]
zone = europe-west2-a
region = europe-west2

The credentials are stored in a sqlite database

GCLOUD- uninstall

This removes the cli and all configuration:

rm -rf ~/cloud/gcp-cli/google-cloud-sdk ~/.config/gcloud

Microsoft – Azure

AZ- install

( rm -rf ~/cloud/azure-cli ; touch ~/cloud/.bashrc; mkdir -p ~/cloud/azure-cli && cd /var/tmp && wget -qc https://aka.ms/InstallAzureCli && sh $(basename $_) && rm -f /var/tmp/InstallAzureCli )
~/cloud/azure-cli
~/cloud/azure-cli
y
~/cloud/.bashrc

I’ve copy-pasted the answers. I didn’t look at the way they do that but I’ve found no obvious way to pass them as parameters or here file. And given how those scripts works I don’t want them to touch my .bashrc file to I do it in one I create in ~/cloud.

alias az || echo 'alias az=~/cloud/azure-cli/az' >> ~/.bashrc && . ~/.bashrc
az version

AZ- update

az upgrade

AZ- config

$ az login

The authentication gives you an URL where you allow paste a code and the authentication is automatic. The info and credential tokens are stored in ~/.azure/azureProfile.json ~/.azure/accessTokens.json

AZ- uninstall

rm -rf ~/cloud/azure-cli

Oracle – OCI

OCI – install

( mkdir -p ~/cloud/oci-cli ; cd /var/tmp && wget -qcO oci-cli-install.sh https://raw.githubusercontent.com/oracle/oci-cli/master/scripts/install/install.sh && sh oci-cli-install.sh --install-dir ~/cloud/oci-cli --exec-dir ~/cloud/oci-cli --script-dir ~/cloud/oci-cli --optional-features db --rc-file-path ~/cloud/.bashrc --accept-all-defaults && rm oci-cli-install.sh )

As I did with the previous one I don’t let it change my .bashrc for autocompletion but do it in ~/cloud/.bashrc

alias oci || echo 'alias oci=~/cloud/oci-cli/oci' >> ~/.bashrc && . ~/.bashrc
oci --version


OCI – update

sudo yum install -y python-pip & pip install oci-cli --upgrade

OCI – config

Here you will need to generate a key and get multiple identifiers:

oci setup config

The tool displays a link to the documentation. Better follow it as the console sometimes changes.
Basically, as in the screenshot above, you will need the default region identifier (which you find in the url), the tenant OCID and the user OCID (for which you will add the API key) – can can find those from “profile” upper-right menu.

OCI – uninstall

pip uninstall oci-cli

    $HOME/lib/oracle-cli
    $HOME/bin/oci
    $HOME/bin/oci-cli-scripts

L’article Cloud CLI est apparu en premier sur dbi Blog.

.
In a previous post I tried to build a glossary about Amazon DynamoDB terms that look like relational database terms, but with a different technical meaning. Here is more about it. If you work with AWS Databases and frequently switch between DynamoDB and Aurora, or other RDS databases, you may be confused by the same terms used for different meanings.

An index is a redundant structure that is maintained by the database to provide faster and ordered access when querying on a small part of the table. Basically, rather than scanning a table, or a partition, reading all values, and filtering afterwards, you can access to a small part of it that you don’t have to filter too much, and sort afterwards. This small part is a subset of rows and columns, or items and attributes.

Let’s take an example in Aurora with PostgreSQL compatibility in order to explain what is a covering index and a partial index in the relational database vocabulary.

postgres=> c postgres://reader:NWDMCE5xdipIjRrp@hh-pgsql-public.ebi.ac.uk:5432/pfmegrnargs

pfmegrnargs=> ! rm /var/tmp/rna.csv
pfmegrnargs=> copy rna to '/var/tmp/rna.csv' csv header
COPY

pfmegrnargs=> ! du -h /var/tmp/rna.csv
24G     /var/tmp/rna.csv

I’ve downloaded the RNA table, from @RNAcentral, to local csv, about 25 GB.

postgres=> c postgres://postgres:postgres@database-1.cluster-cvlvfe1jv6n5.eu-west-1.rds.amazonaws.com:5432/postgres
You are now connected to database "postgres" as user "postgres".

I’ve created an RDS Aurora with PostgreSQL compatibility (a small db.r6g.large 2 cVPU 16GB RAM)

postgres=> CREATE TABLE rna (id int8 null, upi varchar(30) not null, "timestamp" timestamp null, userstamp varchar(60) null, crc64 bpchar(16) null, len int4 null, seq_short varchar(4000) null, seq_long text null, md5 varchar(64) null, constraint rna_pkey primary key (upi));
CREATE TABLE

Here is the same table as the source, but without any index except the primary key

postgres=> copy rna from '/var/tmp/rna.csv' csv header
COPY
postgres=> vacuum rna;
VACUUM

This loaded the 35 million rows from the CSV.

I’ll query on a small time range, the latest rows from this year:

postgres=> explain (analyze,verbose,buffers) select upi from rna where timestamp > date '2021-01-01';

                                                                 QUERY PLAN
--------------------------------------------------------------------------------------------------------------------------------------------
 Gather  (cost=1000.00..3179976.11 rows=254424 width=14) (actual time=38927.822..43965.577 rows=407181 loops=1)
   Output: upi
   Workers Planned: 2
   Workers Launched: 2
   Buffers: shared hit=1265727 read=1702515
   I/O Timings: read=36.660
   ->  Parallel Seq Scan on public.rna  (cost=0.00..3153533.71 rows=106010 width=14) (actual time=38924.249..43688.330 rows=135727 loops=3)
         Output: upi
         Filter: (rna."timestamp" > '2021-01-01'::date)
         Rows Removed by Filter: 11722942
         Buffers: shared hit=1265727 read=1702515
         I/O Timings: read=36.660
         Worker 0: actual time=38908.360..43841.438 rows=120508 loops=1
           Buffers: shared hit=463939 read=561349
           I/O Timings: read=13.017
         Worker 1: actual time=38936.741..43374.967 rows=172038 loops=1
           Buffers: shared hit=377612 read=564988
           I/O Timings: read=11.924
 Planning Time: 0.112 ms
 Execution Time: 44001.385 ms

Without any index, there’s not other choice than scanning the whole table (or partition if it were partitioned). This is long, but automatically parallelized, so it depends on your instance shape and I/O throughput

postgres=> create index demo_index on rna(timestamp);
CREATE INDEX

This creates an index on the timestamp column, the one that I use in my where clause.

postgres=> explain (analyze,verbose,buffers) select upi from rna where timestamp > date '2021-01-01';

                                                               QUERY PLAN
-----------------------------------------------------------------------------------------------------------------------------------------
 Index Scan using demo_index on public.rna  (cost=0.56..236038.54 rows=254424 width=14) (actual time=0.018..237.076 rows=407181 loops=1)
   Output: upi
   Index Cond: (rna."timestamp" > '2021-01-01'::date)
   Buffers: shared hit=34667 read=28
 Planning Time: 0.172 ms
 Execution Time: 260.792 ms

This is an Index Scan: the time range in my where clause is transformed to an index range scan: use the B-Tree structure to go to the first page for this value and follow the link to the next pages until the end value is reached. Those index entries have a reference to the page in the table where the row is, to get the selected column “upi”. This is ok here (3500 page read for 400000 rows) thanks to the good clustering, but it could be worse. If this is a critical use case, we can do better.

postgres=> create index demo_covering_index on rna(timestamp, upi);
CREATE INDEX

That’s another index where I added the “upi” column to the index.

postgres=> explain (analyze,verbose,buffers) select upi from rna where timestamp > date '2021-01-01';

                                                                     QUERY PLAN

----------------------------------------------------------------------------------------------------------------------------------------------------
 Index Only Scan using demo_covering_index on public.rna  (cost=0.56..9488.99 rows=254424 width=14) (actual time=0.017..57.300 rows=407181 loops=1)
   Output: upi
   Index Cond: (rna."timestamp" > '2021-01-01'::date)
   Heap Fetches: 0
   Buffers: shared hit=4464
 Planning Time: 0.182 ms
 Execution Time: 80.701 ms

Now, the same access to the index leaves doesn’t have to fetch from the table because the “uid” column is also stored there. This is an Index Only Scan. In database vocabulary, this index is called a covering index. And it has read only 4000 buffers, with more chances to get them from a cache hit.

The DynamoDB term for this is projection. In relational databases, the projection is the operations that filters a subset of the columns from a query. So, exactly the opposite of this one where we store (not filter) a superset (not subset) of the columns to index.

create index demo_partial_index on rna(timestamp, upi) where timestamp > date '2021-01-01';

That’s another index with a where clause. This is called a partial index in SQL databases: not all rows are indexed. Here, for example, a reason can be row lifecycle. For fresh data, we query on specific dates. For older data, larger range where a scan is better (partitioned by year for example).

postgres=> explain (analyze,verbose,buffers) select upi from rna where timestamp > date '2021-01-01';

                                                                    QUERY PLAN

---------------------------------------------------------------------------------------------------------------------------------------------------
 Index Only Scan using demo_partial_index on public.rna  (cost=0.42..8856.78 rows=254424 width=14) (actual time=0.016..52.719 rows=407181 loops=1)
   Output: upi
   Heap Fetches: 0
   Buffers: shared hit=4462
 Planning Time: 0.244 ms
 Execution Time: 76.214 ms

Note that I’m running the same query here. In relational databases, the indexes are transparently maintained and used. You don’t have to query them explicitely. You always query a logical view (the table or a view on it) and the query planner will find the best access path for it. Here, because the index is smaller, the access is cheaper, and this one has been chosen. So, in addition to the Index Only Scan, we benefit from partial indexing. The difference is not huge here for a range scan because a B-Tree index is very efficient already. But for larger tables, the depth of the index is smaller. The important thing is that updates on rows that are out of this partial index do not have the overhead of index maintenance.

The DynamoDB term for this, especially in the case of WHERE … IS NOT NULL, is sparse index. In database concepts, a sparse index is something different. But That’s difficult to explain in PostgreSQL where indexes are always dense.

If you take the same example with AWS Aurora with MySQL compatibility, you will not have the same possibilities. MySQL doesn’t have partial indexes. But if you look at the the primary key, you will not see an index. Because InnoDB actually stores the table in an index structure (like a covering index extended to all columns). Because it is physically ordered, you don’t need an additional index structure. The leaves of the B-Tree are the table and the branches are the index. This index is not dense: you don’t need to index each entry. Only the first value of each table page (leaf block here) is sufficient because everything is ordered. This is what is called a sparse index in databases. Sparse indexes are possible only with primary indexes, the ones that can define the physical organization of the table. But DynamoDB uses the term “sparse” for secondary indexes only, where a attribute can nonexistent (primary key attributes are mandatory) in an item, and then not indexed, in the same way as what relational databases call partial indexes.

RDBMS and NoSQL, often presented as opposite, have many similarities: you can store JSON documents in RDS, hash partition those tables on their primary key, and you have a NoSQL data structure. And you can query the NoSQL databases with an API that looks like SQL (PartiQL for example). And I think that the same converged data platform could be used by both APIs, in order to de-correlate the microservices isolation from the distributed infrastructure. Actually the storage design of Aurora and DynamoDB have some similarities. The big difference is in what NoSQL calls “eventual consistency” but that’s for a next post. The difference of vocabulary is really misleading and it starts with NoSQL articles using the term “SQL” as an umbrella for strong typing, relational modeling, declarative language query, ACID properties,… So, the most important is to understand the concepts behing those terms. In summary:

• A projection in RDS is restricting the columns that are read by the SELECT
• A projection in DynamoDB is adding more columns to the index to avoid a table access per item
• A covering index in RDS is adding more columns to the index, with no need to sort on them, and to avoid a table access
• A partial index in RDS is maintaining index entries only for a subset of the rows, to get a smaller B-Tree
• A sparse index in DynamoDB is partially indexing by bypassing secondary index entries for nonexistent attributes
• A sparse index in RDS is avoiding a dense primary index entries thanks to a physically ordered table

L’article DynamoDB / Aurora: sparse and partial indexes est apparu en premier sur dbi Blog.

.
If you are surprised that the first thing you do in a Public Cloud is creating a Virtual Private Cloud, this post is for you. This is a beginner level post. And if you are at that level, interested by what is the Cloud and what is AWS, I recommend our free AWS Discovery Days – I give it next week in French: https://www.dbi-services.com/fr/trainings/aws-discovery-days/

Today, with the “digitalization” trend, people are looking at the cloud as a way to move their IT from their own data centers to a public cloud provider. And then, the first thing they will try to understand is how their current infrastructure matches in cloud terms. Everything runs in virtual machines (VM) on their self managed hypervisor. And the equivalent in public clouds is there with compute services: Amazon EC2, Google VM instances, Azure Virtual Machines, Oracle Compute Instances. Different names for virtual computers. And, besides defining their number of vCPUs and amount of RAM (Instance Type in AWS, Machine Type in Google, VM Size in Azure, Instance Shape in Oracle) and before attaching it to disks (block storage), the first thing to do is define on which network it is connected. This network is called VPC (Virtual Private Cloud) in AWS and Google, Virtual Network in Azure, VCN (Virtual Cloud Network) in Oracle. On-premises you have VLANs and subnets. And you can have private cloud. So why is this called Virtual Private Cloud in a Public Cloud?

Actually, you will never see Amazon talking about AWS as a Public Cloud provider. It is a cloud provider with public services, but today what you run in a public cloud can run as a private cloud: AWS Outpost, Google Anthos, Azure Stack, Oracle Cloud@Customer…

And, first of all, the most “cloudy” services in AWS do not run in a VPC at all. These days, Amazon is celebrating the 15 years of its first service: S3 (Simple Storage Service). This didn’t match with anything existing on-premises. Not because of the Web API (HTTP, FTP WebDAV where already there, and Oracle even had iFS at that time). But because it is a public service: no need to provision a network, and servers. It runs over the Internet. That’s where AWS comes from: Amazon Web Services. You want to store a file and access it? Before, you needed a server. Now, you have a service. That’s what “serverless” means. Of course there are servers, but you don’t know were they are, how much they cost, how many of them… You interact only with a service. And today, being truely “cloud-native” and “serverless” is about having files in S3, code in Lambda, data in DynamoDB, interfaces in SQS, messages in SNS… None of those services requires a VPC. Their endpoint is a public IP address on the Internet.

Now you understand how we can have a Private Cloud in a Public Cloud. For the things that are not serverless, where you must have a dedicated VM, you need virtual servers and virtual network, as in your private cloud. When you create an EC2 instance, you do the same as when you create a VMware VM on your premises, except that it is hosted in a cloud provider’s datacenter, and can share some of its infrastructure.

The data center is, physically, in an Availability Zone. For High Availability, there are multiple Availability Zones in a Region. And for Disaster Recovery, latency or data locality, or simply because of price, you have multiple regions in the world. A VPC belongs to a region and has subnets in the Availability Zones. They are defined by their network identification, which is the CIDR. For example, A VPC in /16 has 16 bits to identify it: the first two bytes are fixed, and the subnet mask is 255.255.0.0 and all the 65536 addresses in this VPC are your addresses (except a few used by the infrastructure itself). Virtual doesn’t mean that they are isolated. One day you may want to peer two VPCs across different accounts or regions and then you must be sure that the CIDR ranges do not overlap. You deploy subnets to put VMs in specific availability zones, isolate logically your components, and isolate their services. Even if you can manage the access to the network services at instance level with Security Groups, you can allow or deny trafic from/to your subnet with NACLs. And you have access to the route table in order to define what and how you can reach other networks. For example, a public facing subnet will route though an Internet Gateway. And if you want to access the public cloud services, like S3 or DynamoDB, you go though the internet gateway, or a gateway endpoint which can expose the internet service without going though the internet.

So, even if you first encounter with AWS is though a VPC you should keep in mind that many services do not belong to a VPC and many do not belong to a region either. Take S3 for example. There’s no VPC. The endpoint is on Internet. And the service is not even regional. You create the bucket in a region, because you want it near your users, but the service is Global. For data, DynamoDB and IoT Core are serverless. But databases still need a server with CPU and RAM. Even Aurora Serverless is running in a VPC. The database server is managed for you, and may be re-sized, and scaled with read replicas. But there’s no relational database that runs of of a VPC.

L’article A VPC is a private cloud in a public cloud est apparu en premier sur dbi Blog.

.
I have posted, a long time ago, about Google Spanner (inserting data and no decimal numeric data types) but many things have changed in this area. There is now a NUMERIC data type and many things have improved in this distributed SQL database, improving a bit the SQL compatibility.

gcloud

I can use the Cloud Shell, which is very easy – one click fron the console – but here I’m showing how to install the gcloud CLI temporarily in a OEL environement (I’ve explained in a previous post how the OCI free tier is my preferred home)

curl https://dl.google.com/dl/cloudsdk/channels/rapid/downloads/google-cloud-sdk-321.0.0-linux-x86_64.tar.gz | tar -C /var/tmp -zxf -
/var/tmp/google-cloud-sdk/install.sh --usage-reporting true --screen-reader false --rc-path /tmp/gcloud.tmp --command-completion true --path-update true --override-components

This downloads and unzips the Google Cloud SDK for Linux (there are other options like a YUM repo). I put it in a temporary directory here under /var/tmp. install.sh is interactive or you can supply all information on command line. I don’t want it to update my .bash_profile or .bashrc but want to see what it puts there, so just providing a temporary /tmp/gcloud.sh to have a lookt at it

[opc@a ~]$ /var/tmp/google-cloud-sdk/install.sh --usage-reporting true --screen-reader false --rc-path /tmp/gcloud.sh --command-completion true --path-update true --override-components

Welcome to the Google Cloud SDK!

Your current Cloud SDK version is: 321.0.0
The latest available version is: 321.0.0

┌────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│                                                 Components                                                 │
├───────────────┬──────────────────────────────────────────────────────┬──────────────────────────┬──────────┤
│     Status    │                         Name                         │            ID            │   Size   │
├───────────────┼──────────────────────────────────────────────────────┼──────────────────────────┼──────────┤
...
│ Installed     │ BigQuery Command Line Tool                           │ bq                       │  < 1 MiB │
│ Installed     │ Cloud SDK Core Libraries                             │ core                     │ 15.9 MiB │
│ Installed     │ Cloud Storage Command Line Tool                      │ gsutil                   │  3.5 MiB │
└───────────────┴──────────────────────────────────────────────────────┴──────────────────────────┴──────────┘

[opc@a ~]$ cat /tmp/gcloud.tmp

# The next line updates PATH for the Google Cloud SDK.
if [ -f '/var/tmp/google-cloud-sdk/path.bash.inc' ]; then . '/var/tmp/google-cloud-sdk/path.bash.inc'; fi

# The next line enables shell command completion for gcloud.
if [ -f '/var/tmp/google-cloud-sdk/completion.bash.inc' ]; then . '/var/tmp/google-cloud-sdk/completion.bash.inc'; fi

As you can see, I mentioned nothing for –override-components and the default is gsutil, core and bq

This is the most simple cloud CLI I ever seen. Just type: gcloud init
(/var/tmp/google-cloud-sdk/bin/gcloud in my temporary installation) and it gives you an URL where you can get the verification code, using the web console authentication. Then you pick the cloud project and, optionally, a default region and zone. Those default informations are stored in: ~/.config/gcloud/configurations/config_default
and the credentials are in a sqllite database in ~/.config/gcloud/credentials.db

Sample SQL data

Gerald Venzl has recently published some free to use data about the world’s countries, capitals, and currencies. What I like with this data set is that, in addition to providing the CSV, Gerald have managed to provide a unique DDL + DML to create this data in SQL databases. And this works in the most common databases despite the fact that, beyond the SQL standard, data types and syntax is different in each engine.

Christmas comes early for #data geeks! Thanks to the help of my lovely wife, I present you with #free to use, @creativecommons licensed data about our world's countries, capitals, and currencies! We hope you enjoy and wish you Happy Holidays!https://t.co/zsRpZ4PXP2 pic.twitter.com/yQDTmB6BMv

— Gerald Venzl (@GeraldVenzl) December 14, 2020

Google Spanner has a SQL-like API but I cannot run this without a few changes. But just a few, thanks to many recent improvements. And loading this data set will be the occasion to show those new features.

Primary key

Here is the DDL to create the first table, REGIONS:

/*********************************************/
/***************** REGIONS *******************/
/*********************************************/
CREATE TABLE regions
(
  region_id     VARCHAR(2)   NOT NULL,
  name          VARCHAR(13)  NOT NULL,
  CONSTRAINT regions_pk
    PRIMARY KEY (region_id)
);

In order to run this in Google Spanner, I change VARCHAR to STRING and I move the PRIMARY KEY declaration out of the relational properties:

CREATE TABLE regions
(
  region_id     STRING(2)   NOT NULL,
  name          STRING(13)  NOT NULL
)
    PRIMARY KEY (region_id)
;

It may be surprising to have the PRIMARY KEY declaration at this place but, because Google Spanner is a distributed database, the primary key is also a storage attribute. And it is mandatory as sharding is done by range partitioning on the primary key. Well, I would prefer to stay compatible with SQL and have, if needed, an additional organization clause. But Spanner is one of the first database trying to bring SQL to NoSQL and was originally designed to be used internally (like Google object storage matadata), providing SQL database benefits (SQL, ACID, joins,…) with the same scalability as distributed NoSQL databases. So compatibility with other SQL databases was probably not a priority.

Note that the NOT NULL constraint is allowed and we will see more about columns constraints later.

I have removed the comments because this is not allowed in Google Spanner. I don’t understand that, but remember that it takes its roots in NoSQL where the API calls are embedded in the code, and not in scripts, and the code has its own comments.

Foreign key

The second table is COUNTRIES and a country belongs to a region from the REGION table.

/*********************************************/
/**************** COUNTRIES ******************/
/*********************************************/
CREATE TABLE countries
(
  country_id    VARCHAR(3)     NOT NULL,
  country_code  VARCHAR(2)     NOT NULL,
  name          VARCHAR(100)   NOT NULL,
  official_name VARCHAR(200),
  population    NUMERIC(10),
  area_sq_km    NUMERIC(10,2),
  latitude      NUMERIC(8,5),
  longitude     NUMERIC(8,5),
  timezone      VARCHAR(40),
  region_id     VARCHAR(2)     NOT NULL,
  CONSTRAINT countries_pk
    PRIMARY KEY (country_id),
  CONSTRAINT countries_regions_fk001
    FOREIGN KEY (region_id) REFERENCES regions (region_id)
);

CREATE INDEX countries_regions_fk001 ON countries (region_id);

On the datatypes, I’ll change VARCHAR to STRING and now we have a NUMERIC datatype in Spanner (was only IEEE 754 float) but NUMERIC has a fixed scale and precision (38,9) I’ll probably come back to it in another post. But there are still many limitations with NUMERIC (cannot create index on it, not easy to map when importing,…). It is definitely not a good choice here for areas, latitude and longitude. But I keep it just for DDL compatibility.

I move the PRIMARY KEY definition. But the most important here is actually about not changing the referential constraint at all:

CREATE TABLE countries
(
  country_id    STRING(3)     NOT NULL,
  country_code  STRING(2)     NOT NULL,
  name          STRING(100)   NOT NULL,
  official_name STRING(200),
  population    NUMERIC,
  area_sq_km    NUMERIC,
  latitude      NUMERIC,
  longitude     NUMERIC,
  timezone      STRING(40),
  region_id     STRING(2)     NOT NULL,
  CONSTRAINT countries_regions_fk001
    FOREIGN KEY (region_id) REFERENCES regions (region_id)
)
PRIMARY KEY (country_id)
;

Before March 2020 The only possible referential integrity way actually a storage clause (at the same place as the PRIMARY KEY declaration) because referential integrity is not something easy in a distributed database. Because you distribute to scale and that works well only when your transaction is single-shard. This is why, initially, referential integrity was not a FOREIGN KEY but a compound PRIMARY KEY interleaved with the parent PRIMARY KEY. But we will see INTERLEAVE IN PARENT later. Here, COUTRIES has its own primary key, and then its own sharding scheme. That’ also mean that inserting a new country may have to check the parent key (region) in another shard. We will look at performance in another post.

When we declare a foreign key in Google Spanner, an index on it is created, then I didn’t copy the CREATE INDEX statement.

Check constraints

/*********************************************/
/***************** CITIES ********************/
/*********************************************/

CREATE TABLE cities
(
  city_id       VARCHAR(7)    NOT NULL,
  name          VARCHAR(100)  NOT NULL,
  official_name VARCHAR(200),
  population    NUMERIC(8),
  is_capital    CHAR(1)       DEFAULT 'N' NOT NULL,
  latitude      NUMERIC(8,5),
  longitude     NUMERIC(8,5),
  timezone      VARCHAR(40),
  country_id    VARCHAR(3)    NOT NULL,
  CONSTRAINT cities_pk
    PRIMARY KEY (city_id),
  CONSTRAINT cities_countries_fk001
    FOREIGN KEY (country_id) REFERENCES countries (country_id),
  CONSTRAINT cities_is_capital_Y_N_check001
    CHECK (is_capital IN ('Y','N'))
);

CREATE INDEX cities_countries_fk001 ON cities (country_id);

In addition to the datatypes we have seen earlier I’ll transform CHAR to STRING, but I have to remove the DEFAULT declaration. Spanner recently introduced generated columns but those are always calculated, they cannot substitute to DEFAULT.
We declare check constraints since Oct. 2020 and I keep the same declaration. As far as I know they are not used by the optimizer but only to validate the data ingested.

I have a foreign key to COUNTRIES which I keep as non-interleaved. Because the COUNTRY_ID is not part of the CITIES primary key, and maybe because my data model may have to cope with cities changing to another country (geopolitical immutability).

CREATE TABLE cities
(
  city_id       STRING(7)    NOT NULL,
  name          STRING(100)  NOT NULL,
  official_name STRING(200),
  population    NUMERIC,
  is_capital    STRING(1)       NOT NULL,
  latitude      NUMERIC,
  longitude     NUMERIC,
  timezone      STRING(40),
  country_id    STRING(3)    NOT NULL,
  CONSTRAINT cities_countries_fk001
    FOREIGN KEY (country_id) REFERENCES countries (country_id),
  CONSTRAINT cities_is_capital_Y_N_check001
    CHECK (is_capital IN ('Y','N'))
)
    PRIMARY KEY (city_id)
;

Again, the index on COUNTRY_ID is created implicitely.

Interleaved

/*********************************************/
/***************** CURRENCIES ****************/
/*********************************************/
CREATE TABLE currencies
(
  currency_id       VARCHAR(3)    NOT NULL,
  name              VARCHAR(50)   NOT NULL,
  official_name     VARCHAR(200),
  symbol            VARCHAR(18)   NOT NULL,
  CONSTRAINT currencies_pk
    PRIMARY KEY (currency_id)
);
/*********************************************/
/*********** CURRENCIES_COUNTRIES ************/
/*********************************************/
CREATE TABLE currencies_countries
(
  currency_id    VARCHAR(3)   NOT NULL,
  country_id     VARCHAR(3)   NOT NULL,
  CONSTRAINT currencies_countries_pk
    PRIMARY KEY (currency_id, country_id),
  CONSTRAINT currencies_countries_currencies_fk001
    FOREIGN KEY (currency_id) REFERENCES currencies (currency_id),
  CONSTRAINT currencies_countries_countries_fk002
    FOREIGN KEY (country_id)  REFERENCES countries(country_id)
);

The CURRENCIES_COUNTRIES is the implementation of many-to-many relationship as a country may have multiple currencies and a currency can be used in multiple country. The primary key is a concatenation of the foreign keys. Here I’ll decide that referential integrity is a bit stronger and the list of currencies will be stored in each country, like storing it pre-joined for performance reason. If you come from Oracle, you may see this INTERLEAVE IN PARENT as a table CLUSTER but on a partitioned table. If you come from DynamoDB you may see the of it as the Adjacency lists modeling in the single-table design.

CREATE TABLE currencies
(
  currency_id       STRING(3)    NOT NULL,
  name              STRING(50)   NOT NULL,
  official_name     STRING(200),
  symbol            STRING(18)   NOT NULL,
)
    PRIMARY KEY (currency_id)
;

CREATE TABLE currencies_countries
(
  currency_id    STRING(3)   NOT NULL,
  country_id     STRING(3)   NOT NULL,
  CONSTRAINT currencies_countries_countries_fk002
    FOREIGN KEY (country_id)  REFERENCES countries(country_id)
)
PRIMARY KEY (currency_id, country_id)
,  INTERLEAVE IN PARENT currencies ON DELETE CASCADE;

Here, the many-to-many association between CURRENCIES and COUNTRIES is materialized as a composition with CURRENCIES, stored with it (INTERLEAVE), and removed with it (ON DELETE CASCADE). Note that I did that for the demo because CURRENCY_ID was first in the primary key declaration, but you may think more about data distribution and lifecycle when deciding on interleaving. Google Spanner partitions by range, and this means that all CURRENCIES_COUNTRIES associations will be stored together, in the same shard (called “split” in Spanner) for the same CURRENCY_ID.

Multi-region instance

There are many new multi-region configurations, within the same continent or distributed over multiple ones:

gcloud spanner instance-configs list

NAME                              DISPLAY_NAME
asia1                             Asia (Tokyo/Osaka/Seoul)
eur3                              Europe (Belgium/Netherlands)
eur5                              Europe (London/Belgium/Netherlands)
eur6                              Europe (Netherlands, Frankfurt)
nam-eur-asia1                     United States, Europe, and Asia (Iowa/Oklahoma/Belgium/Taiwan)
nam10                             United States (Iowa/Salt Lake/Oklahoma)
nam11                             United States (Iowa, South Carolina, Oklahoma)
nam3                              United States (Northern Virginia/South Carolina)
nam6                              United States (Iowa/South Carolina/Oregon/Los Angeles)
nam7                              United States (Iowa, Northern Virginia, Oklahoma)
nam8                              United States (Los Angeles, Oregon, Salt Lake City)
nam9                              United States (Northern Virginia, Iowa, South Carolina, Oregon)
regional-asia-east1               asia-east1
regional-asia-east2               asia-east2
regional-asia-northeast1          asia-northeast1
regional-asia-northeast2          asia-northeast2
regional-asia-northeast3          asia-northeast3
regional-asia-south1              asia-south1
regional-asia-southeast1          asia-southeast1
regional-asia-southeast2          asia-southeast2
regional-australia-southeast1     australia-southeast1
regional-europe-north1            europe-north1
regional-europe-west1             europe-west1
regional-europe-west2             europe-west2
regional-europe-west3             europe-west3
regional-europe-west4             europe-west4
regional-europe-west6             europe-west6
regional-northamerica-northeast1  northamerica-northeast1
regional-southamerica-east1       southamerica-east1
regional-us-central1              us-central1
regional-us-east1                 us-east1
regional-us-east4                 us-east4
regional-us-west1                 us-west1
regional-us-west2                 us-west2
regional-us-west3                 us-west3
regional-us-west4                 us-west4

I’ll use the latest dual-region in my continent, eur6, added on Dec. 2020 which has two read-write regions (Netherlands, Frankfurt) and the witness region in Zurich. Yes, this neutral position of Switzerland is a perfect fit in the distributed quorum, isn’t it?

time gcloud spanner instances create franck --config eur6 --nodes=3 --description Franck

Creating instance...done.

real    0m5.128s
user    0m0.646s
sys     0m0.079s

The instance is created, and we can create multiple database in it (there is no schemas, so database is the right logical isolation between database objects)

time gcloud spanner databases create test --instance=franck --ddl "
CREATE TABLE regions
(
  region_id     STRING(2)   NOT NULL,
  name          STRING(13)  NOT NULL
)
    PRIMARY KEY (region_id)
;
CREATE TABLE countries
(
  country_id    STRING(3)     NOT NULL,
  country_code  STRING(2)     NOT NULL,
  name          STRING(100)   NOT NULL,
  official_name STRING(200),
  population    NUMERIC,
  area_sq_km    NUMERIC,
  latitude      NUMERIC,
  longitude     NUMERIC,
  timezone      STRING(40),
  region_id     STRING(2)     NOT NULL,
  CONSTRAINT countries_regions_fk001
    FOREIGN KEY (region_id) REFERENCES regions (region_id)
)
PRIMARY KEY (country_id)
;
CREATE TABLE cities
(
  city_id       STRING(7)    NOT NULL,
  name          STRING(100)  NOT NULL,
  official_name STRING(200),
  population    NUMERIC,
  is_capital    STRING(1)       NOT NULL,
  latitude      NUMERIC,
  longitude     NUMERIC,
  timezone      STRING(40),
  country_id    STRING(3)    NOT NULL,
  CONSTRAINT cities_countries_fk001
    FOREIGN KEY (country_id) REFERENCES countries (country_id),
  CONSTRAINT cities_is_capital_Y_N_check001
    CHECK (is_capital IN ('Y','N'))
)
    PRIMARY KEY (city_id)
;
CREATE TABLE currencies
(
  currency_id       STRING(3)    NOT NULL,
  name              STRING(50)   NOT NULL,
  official_name     STRING(200),
  symbol            STRING(18)   NOT NULL,
)
    PRIMARY KEY (currency_id)
;
CREATE TABLE currencies_countries
(
  currency_id    STRING(3)   NOT NULL,
  country_id     STRING(3)   NOT NULL,
  CONSTRAINT currencies_countries_countries_fk002
    FOREIGN KEY (country_id)  REFERENCES countries(country_id)
)
PRIMARY KEY (currency_id, country_id)
,  INTERLEAVE IN PARENT currencies ON DELETE CASCADE;
"

ERROR: (gcloud.spanner.databases.create) INVALID_ARGUMENT: Error parsing Spanner DDL statement: \n : Syntax error on line 1, column 1: Encountered \'EOF\' while parsing: ddl_statement
- '@type': type.googleapis.com/google.rpc.LocalizedMessage
  locale: en-US
  message: |-
    Error parsing Spanner DDL statement:
     : Syntax error on line 1, column 1: Encountered 'EOF' while parsing: ddl_statement

No idea why it didn’t parse my DDL. The same in the console works, so this is how I’ve created those tables. Now re-reading this and the problem may be only that I start with an newline here before the first CREATE.

My database is ready for DDL and DML, provisioned in a few seconds.

Foreign Key indexes

You mave seen that I removed the CREATE INDEX that was in the original DDL. This is because an index is created automatically on the foreign key (which makes sense, we want to avoid full table scans at all prize in a distributed database).

gcloud spanner databases ddl update test --instance=franck --ddl="CREATE INDEX countries_regions_fk001 ON countries (region_id)"

Schema updating...failed.
ERROR: (gcloud.spanner.databases.ddl.update) Duplicate name in schema: countries_regions_fk001.

Trying to create an index with the same name as the foreign key is rejected as “duplicate name”. However, when looking at the ddl:

gcloud spanner databases ddl describe test --instance=franck | grep -i "CREATE INDEX"

Shows no index. And trying to query explicitly through this index with this name is rejected:

gcloud spanner databases execute-sql test --instance=franck --query-mode=PLAN \
 --sql "select count(region_id) from countries@{FORCE_INDEX=countries_regions_fk001}"

ERROR: (gcloud.spanner.databases.execute-sql) INVALID_ARGUMENT: The table countries does not have a secondary index called countries_regions_f
k001

However this index exists, with another name. Let’s do an explain plan when querying this column only:

gcloud spanner databases execute-sql test --instance=franck --query-mode=PLAN \
 --sql "select count(region_id) from countries"

 RELATIONAL Serialize Result
    |
    +- RELATIONAL Aggregate
    |  call_type: Global, iterator_type: Stream, scalar_aggregate: true
    |   |
    |   +- RELATIONAL Distributed Union
    |   |  subquery_cluster_node: 3
    |   |   |
    |   |   +- RELATIONAL Aggregate
    |   |   |  call_type: Local, iterator_type: Stream, scalar_aggregate: true
    |   |   |   |
    |   |   |   +- RELATIONAL Distributed Union
    |   |   |   |  call_type: Local, subquery_cluster_node: 5
    |   |   |   |   |
    |   |   |   |   \- RELATIONAL Scan
    |   |   |   |      Full scan: true, scan_target: IDX_countries_region_id_1D2B1A686087F93F, scan_type: IndexScan
    |   |   |   |
    |   |   |   \- SCALAR Function
    |   |   |      COUNT(1)
    |   |   |       |
    |   |   |       \- SCALAR Constant
    |   |   |          1
    |   |   |
    |   |   \- SCALAR Constant
    |   |      true
    |   |
    |   \- SCALAR Function
    |      COUNT_FINAL($v1)
    |       |
    |       \- SCALAR Reference
    |          $v1
    |
    \- SCALAR Reference
       $agg1

The foreign key index name is: IDX_countries_region_id_1D2B1A686087F93F

gcloud spanner databases execute-sql test --instance=franck --query-mode=PLAN \
 --sql "select count(region_id) from countries@{FORCE_INDEX=IDX_countries_region_id_1D2B1A686087F93F}" \
 | grep scan

    |   |   |   |      Full scan: true, scan_target: IDX_countries_region_id_1D2B1A686087F93F, scan_type: IndexScan

I can use this index name here. Apparently, the index that is implicitly created with the foreign key has an internal name, but also reserves the constraint name in the index namespace. Let’s validate that the initial error was about the name of the index and not the same columns:

gcloud spanner databases ddl update test --instance=franck \
 --ddl="CREATE INDEX a_silly_redundant_index ON countries (region_id)"

Schema updating...done.

This works, I can create a secondary index on the foreign key columns

gcloud spanner databases execute-sql test --instance=franck --query-mode=PLAN \
 --sql "select count(region_id) from countries"\
 | grep scan

    |   |   |   |      Full scan: true, scan_target: a_silly_redundant_index, scan_type: IndexScan

Querying without mentioning the index shows that it can be used instead of the implicit one. Yes, Google Spanner has now an optimizer that can decide to use a secondary index. But I’ll come back to that in a future blog post. Why using this new index rather than the one created implicitly, as they have the same cost? I don’t know but I’ve started it with an “a” in case alphabetical order wins…

gcloud spanner databases ddl describe test --instance=franck | grep -i "CREATE INDEX"

CREATE INDEX a_silly_redundant_index ON countries(region_id);

gcloud spanner databases ddl update test --instance=franck \
 --ddl="DROP INDEX a_silly_redundant_index"

Schema updating...done.

I remove this index which has no reason to be there.

insert

Executing multiple DML statements at once is not possible. It is possible to insert multiple rows into one table like this:

INSERT INTO regions (region_id, name) VALUES
   ('AF', 'Africa')
 , ('AN', 'Antarctica')
 , ('AS', 'Asia')
 , ('EU', 'Europe')
 , ('NA', 'North America')
 , ('OC', 'Oceania')
 , ('SA', 'South America')
;

However, the initial SQL file doesn’t insert into the same number of columns, like:

INSERT INTO cities (city_id, name, population, is_capital, latitude, longitude, country_id) VALUES ('AFG0001', 'Kabul', 4012000, 'Y', 34.52813, 69.17233, 'AFG');
INSERT INTO cities (city_id, name, official_name, population, is_capital, latitude, longitude, country_id) VALUES ('ATG0001', 'Saint John''s', 'Saint John’s', 21000, 'Y', 17.12096, -61.84329, 'ATG');
INSERT INTO cities (city_id, name, population, is_capital, latitude, longitude, country_id) VALUES ('ALB0001', 'Tirana', 476000, 'Y', 41.3275, 19.81889, 'ALB');

and then a quick replace is not easy. On this example you see something else. There are double quotes, the SQL way to escape quotes, but Spanner doesn’t like it.

Then, without writing a program to insert in bulk, here is what I did to keep the SQL statements from the source:

curl -s https://raw.githubusercontent.com/gvenzl/sample-data/master/countries-cities-currencies/install.sql \
 | awk "/^INSERT INTO /{gsub(/''/,quote);print}" quote="\\\\\\\'" \
 | while read sql ; do echo "$sql" ; gcloud spanner databases execute-sql test --instance=franck --sql="$sql" ; done \
 | ts | tee insert.log

I just replaced the quote escaping ” by //. It is slow row-by-row but all rows are inserted with the original SQL.

I have 3 nodes running, two in read-write in two regions (europe-west3 is “Frankfurt”, europe-west4 is “Netherlands”) and one acting as witness for the quorum (europe-west6 in Zürich).

select

With the sample data provided by Gerald, there’s a query to check the number of rows:

gcloud spanner databases execute-sql test --instance=franck --query-mode=NORMAL --sql "SELECT 'regions' AS Table, 7 AS provided,
COUNT(1) AS actual FROM regions                                                                                                               
UNION ALL
SELECT 'countries' AS Table, 196 AS provided, COUNT(1) AS actual FROM countries
UNION ALL
SELECT 'cities' AS Table, 204 AS provided, COUNT(1) AS actual FROM cities
UNION ALL
SELECT 'currencies' AS Table, 146 AS provided, COUNT(1) AS actual FROM currencies
UNION ALL
SELECT 'currencies_countries' AS Table, 203 AS provided, COUNT(1) AS actual FROM currencies_countries;
"

Table                 provided  actual
regions               7         7
countries             196       196
cities                204       204
currencies            146       146
currencies_countries  203       203

The result is correct. By curiosity, I’m running it with execution plan and statistics (query mode “PROFILE”):

gcloud spanner databases execute-sql test --instance=franck --query-mode=PROFILE --sql "SELECT 'regions' AS Table, 7 AS provided,
 COUNT(1) AS actual FROM regions
 UNION ALL
 SELECT 'countries' AS Table, 196 AS provided, COUNT(1) AS actual FROM countries
 UNION ALL
 SELECT 'cities' AS Table, 204 AS provided, COUNT(1) AS actual FROM cities
 UNION ALL
 SELECT 'currencies' AS Table, 146 AS provided, COUNT(1) AS actual FROM currencies
 UNION ALL
 SELECT 'currencies_countries' AS Table, 203 AS provided, COUNT(1) AS actual FROM currencies_countries;
 "

┌────────────────────┬────────────┬───────────────┬──────────────┬───────────────────┐
│ TOTAL_ELAPSED_TIME │  CPU_TIME  │ ROWS_RETURNED │ ROWS_SCANNED │ OPTIMIZER_VERSION │
├────────────────────┼────────────┼───────────────┼──────────────┼───────────────────┤
│ 14.7 msecs         │ 4.92 msecs │ 5             │ 756          │ 2                 │
└────────────────────┴────────────┴───────────────┴──────────────┴───────────────────┘
 RELATIONAL Serialize Result
 (1 execution, 0.45 msecs total latency)
    |
    +- RELATIONAL Union All
    |  (1 execution, 0.44 msecs total latency)
    |   |
    |   +- RELATIONAL Union Input
    |   |   |
    |   |   +- RELATIONAL Compute
    |   |   |   |
    |   |   |   +- RELATIONAL Aggregate
    |   |   |   |  (1 execution, 0.06 msecs total latency)
    |   |   |   |  call_type: Global, iterator_type: Stream, scalar_aggregate: true
    |   |   |   |   |
    |   |   |   |   +- RELATIONAL Distributed Union
    |   |   |   |   |  (1 execution, 0.06 msecs total latency)
    |   |   |   |   |  subquery_cluster_node: 6
    |   |   |   |   |   |
    |   |   |   |   |   +- RELATIONAL Aggregate
    |   |   |   |   |   |  (1 execution, 0.04 msecs total latency)
    |   |   |   |   |   |  call_type: Local, iterator_type: Stream, scalar_aggregate: true
    |   |   |   |   |   |   |
    |   |   |   |   |   |   +- RELATIONAL Distributed Union
    |   |   |   |   |   |   |  (1 execution, 0.04 msecs total latency)
    |   |   |   |   |   |   |  call_type: Local, subquery_cluster_node: 8
    |   |   |   |   |   |   |   |
    |   |   |   |   |   |   |   \- RELATIONAL Scan
    |   |   |   |   |   |   |      (1 execution, 0.03 msecs total latency)
    |   |   |   |   |   |   |      Full scan: true, scan_target: regions, scan_type: TableScan
    |   |   |   |   |   |   |
    |   |   |   |   |   |   \- SCALAR Function
    |   |   |   |   |   |      COUNT(1)
    |   |   |   |   |   |       |
    |   |   |   |   |   |       \- SCALAR Constant
    |   |   |   |   |   |          1
    |   |   |   |   |   |
    |   |   |   |   |   \- SCALAR Constant
    |   |   |   |   |      true
    |   |   |   |   |
    |   |   |   |   \- SCALAR Function
    |   |   |   |      COUNT_FINAL($v1)
    |   |   |   |       |
    |   |   |   |       \- SCALAR Reference
    |   |   |   |          $v1
    |   |   |   |
    |   |   |   +- SCALAR Constant
    |   |   |   |  'regions'
    |   |   |   |
    |   |   |   \- SCALAR Constant
    |   |   |      7
    |   |   |
    |   |   +- SCALAR Reference
    |   |   |  $Table
    |   |   |
    |   |   +- SCALAR Reference
    |   |   |  $provided
    |   |   |
    |   |   \- SCALAR Reference
    |   |      $actual
    |   |
    |   +- RELATIONAL Union Input
    |   |   |
    |   |   +- RELATIONAL Compute
    |   |   |   |
    |   |   |   +- RELATIONAL Aggregate
    |   |   |   |  (1 execution, 0.1 msecs total latency)
    |   |   |   |  call_type: Global, iterator_type: Stream, scalar_aggregate: true
    |   |   |   |   |
    |   |   |   |   +- RELATIONAL Distributed Union
    |   |   |   |   |  (1 execution, 0.1 msecs total latency)
    |   |   |   |   |  subquery_cluster_node: 23
    |   |   |   |   |   |
    |   |   |   |   |   +- RELATIONAL Aggregate
    |   |   |   |   |   |  (1 execution, 0.09 msecs total latency)
    |   |   |   |   |   |  call_type: Local, iterator_type: Stream, scalar_aggregate: true
    |   |   |   |   |   |   |
    |   |   |   |   |   |   +- RELATIONAL Distributed Union
    |   |   |   |   |   |   |  (1 execution, 0.09 msecs total latency)
    |   |   |   |   |   |   |  call_type: Local, subquery_cluster_node: 25
    |   |   |   |   |   |   |   |
    |   |   |   |   |   |   |   \- RELATIONAL Scan
    |   |   |   |   |   |   |      (1 execution, 0.08 msecs total latency)
    |   |   |   |   |   |   |      Full scan: true, scan_target: IDX_countries_region_id_1D2B1A686087F93F, scan_type: IndexScan
    |   |   |   |   |   |   |
    |   |   |   |   |   |   \- SCALAR Function
    |   |   |   |   |   |      COUNT(1)
    |   |   |   |   |   |       |
    |   |   |   |   |   |       \- SCALAR Constant
    |   |   |   |   |   |          1
    |   |   |   |   |   |
    |   |   |   |   |   \- SCALAR Constant
    |   |   |   |   |      true
    |   |   |   |   |
    |   |   |   |   \- SCALAR Function
    |   |   |   |      COUNT_FINAL($v2)
    |   |   |   |       |
    |   |   |   |       \- SCALAR Reference
    |   |   |   |          $v2
    |   |   |   |
    |   |   |   +- SCALAR Constant
    |   |   |   |  'countries'
    |   |   |   |
    |   |   |   \- SCALAR Constant
    |   |   |      196
    |   |   |
    |   |   +- SCALAR Reference
    |   |   |  $Table_1
    |   |   |
    |   |   +- SCALAR Reference
    |   |   |  $provided_1
    |   |   |
    |   |   \- SCALAR Reference
    |   |      $actual_1
    |   |
    |   +- RELATIONAL Union Input
    |   |   |
    |   |   +- RELATIONAL Compute
    |   |   |   |
    |   |   |   +- RELATIONAL Aggregate
    |   |   |   |  (1 execution, 0.12 msecs total latency)
    |   |   |   |  call_type: Global, iterator_type: Stream, scalar_aggregate: true
    |   |   |   |   |
    |   |   |   |   +- RELATIONAL Distributed Union
    |   |   |   |   |  (1 execution, 0.11 msecs total latency)
    |   |   |   |   |  subquery_cluster_node: 40
    |   |   |   |   |   |
    |   |   |   |   |   +- RELATIONAL Aggregate
    |   |   |   |   |   |  (1 execution, 0.11 msecs total latency)
    |   |   |   |   |   |  call_type: Local, iterator_type: Stream, scalar_aggregate: true
    |   |   |   |   |   |   |
    |   |   |   |   |   |   +- RELATIONAL Distributed Union
    |   |   |   |   |   |   |  (1 execution, 0.1 msecs total latency)
    |   |   |   |   |   |   |  call_type: Local, subquery_cluster_node: 42
    |   |   |   |   |   |   |   |
    |   |   |   |   |   |   |   \- RELATIONAL Scan
    |   |   |   |   |   |   |      (1 execution, 0.09 msecs total latency)
    |   |   |   |   |   |   |      Full scan: true, scan_target: IDX_cities_country_id_35A7C9365B4BF943, scan_type: IndexScan
    |   |   |   |   |   |   |
    |   |   |   |   |   |   \- SCALAR Function
    |   |   |   |   |   |      COUNT(1)
    |   |   |   |   |   |       |
    |   |   |   |   |   |       \- SCALAR Constant
    |   |   |   |   |   |          1
    |   |   |   |   |   |
    |   |   |   |   |   \- SCALAR Constant
    |   |   |   |   |      true
    |   |   |   |   |
    |   |   |   |   \- SCALAR Function
    |   |   |   |      COUNT_FINAL($v3)
    |   |   |   |       |
    |   |   |   |       \- SCALAR Reference
    |   |   |   |          $v3
    |   |   |   |
    |   |   |   +- SCALAR Constant
    |   |   |   |  'cities'
    |   |   |   |
    |   |   |   \- SCALAR Constant
    |   |   |      204
    |   |   |
    |   |   +- SCALAR Reference
    |   |   |  $Table_2
    |   |   |
    |   |   +- SCALAR Reference
    |   |   |  $provided_2
    |   |   |
    |   |   \- SCALAR Reference
    |   |   |
    |   |   \- SCALAR Reference
    |   |      $actual_2
    |   |
    |   +- RELATIONAL Union Input
    |   |   |
    |   |   +- RELATIONAL Compute
    |   |   |   |
    |   |   |   +- RELATIONAL Aggregate
    |   |   |   |  (1 execution, 0.08 msecs total latency)
    |   |   |   |  call_type: Global, iterator_type: Stream, scalar_aggregate: true
    |   |   |   |   |
    |   |   |   |   +- RELATIONAL Distributed Union
    |   |   |   |   |  (1 execution, 0.08 msecs total latency)
    |   |   |   |   |  subquery_cluster_node: 57
    |   |   |   |   |   |
    |   |   |   |   |   +- RELATIONAL Aggregate
    |   |   |   |   |   |  (1 execution, 0.07 msecs total latency)
    |   |   |   |   |   |  call_type: Local, iterator_type: Stream, scalar_aggregate: true
    |   |   |   |   |   |   |
    |   |   |   |   |   |   +- RELATIONAL Distributed Union
    |   |   |   |   |   |   |  (1 execution, 0.07 msecs total latency)
    |   |   |   |   |   |   |  call_type: Local, subquery_cluster_node: 59
    |   |   |   |   |   |   |   |
    |   |   |   |   |   |   |   \- RELATIONAL Scan
    |   |   |   |   |   |   |      (1 execution, 0.06 msecs total latency)
    |   |   |   |   |   |   |      Full scan: true, scan_target: currencies, scan_type: TableScan
    |   |   |   |   |   |   |
    |   |   |   |   |   |   \- SCALAR Function
    |   |   |   |   |   |      COUNT(1)
    |   |   |   |   |   |       |
    |   |   |   |   |   |       \- SCALAR Constant
    |   |   |   |   |   |          1
    |   |   |   |   |   |
    |   |   |   |   |   \- SCALAR Constant
    |   |   |   |   |      true
    |   |   |   |   |
    |   |   |   |   \- SCALAR Function
    |   |   |   |      COUNT_FINAL($v4)
    |   |   |   |       |
    |   |   |   |       \- SCALAR Reference
    |   |   |   |          $v4
    |   |   |   |          $v4                                                                                                       [49/9050]
    |   |   |   |
    |   |   |   +- SCALAR Constant
    |   |   |   |  'currencies'
    |   |   |   |
    |   |   |   \- SCALAR Constant
    |   |   |      146
    |   |   |
    |   |   +- SCALAR Reference
    |   |   |  $Table_3
    |   |   |
    |   |   +- SCALAR Reference
    |   |   |  $provided_3
    |   |   |
    |   |   \- SCALAR Reference
    |   |      $actual_3
    |   |
    |   +- RELATIONAL Union Input
    |   |   |
    |   |   +- RELATIONAL Compute
    |   |   |   |
    |   |   |   +- RELATIONAL Aggregate
    |   |   |   |  (1 execution, 0.08 msecs total latency)
    |   |   |   |  call_type: Global, iterator_type: Stream, scalar_aggregate: true
    |   |   |   |   |
    |   |   |   |   +- RELATIONAL Distributed Union
    |   |   |   |   |  (1 execution, 0.08 msecs total latency)
    |   |   |   |   |  subquery_cluster_node: 74
    |   |   |   |   |   |
    |   |   |   |   |   +- RELATIONAL Aggregate
    |   |   |   |   |   |  (1 execution, 0.08 msecs total latency)
    |   |   |   |   |   |  call_type: Local, iterator_type: Stream, scalar_aggregate: true
    |   |   |   |   |   |   |
    |   |   |   |   |   |   +- RELATIONAL Distributed Union
    |   |   |   |   |   |   |  (1 execution, 0.07 msecs total latency)
    |   |   |   |   |   |   |  call_type: Local, subquery_cluster_node: 76
    |   |   |   |   |   |   |   |
    |   |   |   |   |   |   |   \- RELATIONAL Scan
    |   |   |   |   |   |   |      (1 execution, 0.07 msecs total latency)
    |   |   |   |   |   |   |      Full scan: true, scan_target: IDX_currencies_countries_country_id_89DC8690B5CAF5C1, scan_type: IndexScan
    |   |   |   |   |   |   |
    |   |   |   |   |   |   \- SCALAR Function
    |   |   |   |   |   |      COUNT(1)
    |   |   |   |   |   |       |
    |   |   |   |   |   |       \- SCALAR Constant
    |   |   |   |   |   |          1
    |   |   |   |   |   |
    |   |   |   |   |   \- SCALAR Constant
    |   |   |   |   |   \- SCALAR Constant                                                                                            [2/9050]
    |   |   |   |   |      true
    |   |   |   |   |
    |   |   |   |   \- SCALAR Function
    |   |   |   |      COUNT_FINAL($v5)
    |   |   |   |       |
    |   |   |   |       \- SCALAR Reference
    |   |   |   |          $v5
    |   |   |   |
    |   |   |   +- SCALAR Constant
    |   |   |   |  'currencies_countries'
    |   |   |   |
    |   |   |   \- SCALAR Constant
    |   |   |      203
    |   |   |
    |   |   +- SCALAR Reference
    |   |   |  $Table_4
    |   |   |
    |   |   +- SCALAR Reference
    |   |   |  $provided_4
    |   |   |
    |   |   \- SCALAR Reference
    |   |      $actual_4
    |   |
    |   +- SCALAR Reference
    |   |  input_0
    |   |
    |   +- SCALAR Reference
    |   |  input_1
    |   |
    |   \- SCALAR Reference
    |      input_2
    |
    +- SCALAR Reference
    |  $Table_5
    |
    +- SCALAR Reference
    |  $provided_5
    |
    \- SCALAR Reference
       $actual_5

Table                 provided  actual

regions               7         7
countries             196       196
cities                204       204
currencies            146       146
currencies_countries  203       203

That’s a long one: scans, joins, concatenation, with some operations run in parallel, projection, aggregation. But basically, it scans each table, or the index when available as we can count the rows from the index which is smaller.

delete instance

This Google Spanner instance costs me $9 per hour (the pricing https://cloud.google.com/spanner/pricing for this eur6 configuration is $3 per node per hour and $50 per 100GB per month). As the provisioning takes 1 minute, and I can copy-paste everything from this blog post, I terminate the instance when I’ve finished my tests.

However, in order to avoid to re-insert those rows, I would be nice to have a backup. Let’s try:

time gcloud spanner backups create sample-data --retention-period=30d --database test --instance=franck
Create request issued for: [sample-data]
Waiting for operation [projects/disco-abacus-161115/instances/franck/backups/sample-data/operations/_auto_op_05f73852e07ae49f] to complete...
⠛
done.
Created backup [sample-data].
real    2m37.593s
user    0m0.908s
sys     0m0.175s

gcloud spanner instances delete franck
Delete instance [franck]. Are you sure?

Do you want to continue (Y/n)?  Y

ERROR: (gcloud.spanner.instances.delete) FAILED_PRECONDITION: Cannot delete instance projects/disco-abacus-161115/instances/franck because it contains backups. Please delete the backups before deleting the instance.

Yes, be careful, a backup is not a database backup here. Just a copy of the tables within the same instance, to be restored to the same region. This is documented of course: Backups reside in the same instance as their source database and cannot be moved. But you know how I find misleading to call that “backup” (a database is living and all the transactions must be backed-up, see What is a database backup). Anyway, this copy cannot help to save money by deleting the instance. The database is gone if I delete the instance and I cannot stop it. The solution is probably to export to the object storage, like a dump, but this will be for a future blog post, maybe.

gcloud spanner backups delete sample-data --instance=franck
You are about to delete backup [sample-data]

Do you want to continue (Y/n)?  Y

Deleted backup [sample-data].

real    0m7.744s
user    0m0.722s
sys     0m0.056s
[opc@a tmp]$ gcloud spanner instances delete franck --quiet   

The –quiet argument bypasses the interactive confirmation.

This test was with small volume, simple schema, from a SQL script that works as-is on the most common SQL databases. Here I had to adapt a few things, but the compatibility with RDBMS has improved a lot in the past year as we can have full referential integrity (foreign key not limited to hierarchical interleave) and logical independence where a secondary index can be used without explicitly mentioning it. Google Spanner is probably the most efficient distributed database when you develop with a NoSQL approach but want to benefit from some SQL features to get a more agile data model and easier integrity and transaction management. However, I don’t think you can port any existing SQL application easily, or even consider it as a general purpose SQL database. For this, you may look at YugaByteDB which uses the PostgreSQL upper layer on distributed database similar to Spanner. However, Spanner has probably the lowest latency for multi-shard operations as it has some cloud-vendor specific optimizations like TrueTime, for software and hardware synchronization.

L’article Google Spanner – SQL compatibility est apparu en premier sur dbi Blog.

.
You may want your RDS database to interact with other AWS services. For example, send a notification on a business or administration situation, with a “push” method rather than a “pull” one from a Cloud watch alert. You may even design this call to be triggered on database changes. And Amazon Aurora provides this possibility by running a lambda from the database through calling mysql.lambda_async() from a MySQL trigger. This is an interesting feature but I think that it is critical to understand how it works in order to use it correctly.
This is the kind of feature that looks very nice on a whiteboard or powerpoint: the DML event (like an update) runs a trigger that calls the lambda, all event-driven. However, this is also dangerous: are you sure that every update must execute this process? What about an update during an application release, or a dump import, or a logical replication target? Now imagine that you have a bug in your application that has set some wrong data and you have to fix it in emergency in the production database, under stress, with manual updates and aren’t aware of that trigger, or just forget about it in this situation… Do you want to take this risk? As the main idea is to run some external service, the consequence might be very visible and hard to fix, like spamming all your users, or involuntarily DDoS a third-tier application.

I highly encourage to encapsulate the DML and the call of lambda in a procedure that is clearly named and described. For example, let’s take a silly example: sending a “your address has been changed” message when a user updates his address. Don’t put the “send message” call in an AFTER UPDATE trigger. Because the UPDATE semantic is to update. Not to send a message. What you can do is write a stored procedure like UPDATE_ADDRESS() that will do the UPDATE, and call the “send message” lambda. You may even provide a boolean parameter to enable or not the sending of the message. Then, the ones who call the stored procedure know what will happen. And the one who just do an update,… will just do an update. Actually, executing DML directly from the application is often a mistake. A database should expose business-related data services, like many other components of your application architecture, and this is exactly the goal of stored procedures.

I’m sharing here some tests on calling lambda from Aurora MySQL.

Wiring the database to lambdas

A lambda is not a simple procedure that you embed in your program. It is a service and you have to control the access to it:

• You create the lambda (create function, deploy and get the ARN)
• You define an IAM policy to invoke this lambda
• You define an IAM role to apply this policy
• You set this role as aws_default_lambda_role in the RDS cluster parameter group
• You add this role to the cluster (RDS -> database cluster -> Manage IAM roles)

Here is my lambda which just logs the event for my test:

import json

def lambda_handler(event, context):
    print('Hello.')
    print(event)
    return {
        'statusCode': 200,
        'body': json.dumps('Hello from Lambda!')
    }

Creating the test database

 drop database if exists demo;
 create database demo;
 use demo;
 drop table if exists t;
 create table t ( x int, y int );
 insert into t values ( 1, 1 );

I have a simple table here, with a simple row.

delimiter $$
create trigger t_bufer before update on t for each row
begin
 set NEW.y=OLD.x;
 call mysql.lambda_async(
    'arn:aws:lambda:eu-central-1:802756008554:function:log-me',
    concat('{"trigger":"t_bufer","connection":"',connection_id(),'","old": "',OLD.x,'","new":"',NEW.x,'"}'));
end;
$$
delimiter ;

This is my trigger which calls my lambda on an update with old and new value in the message.

MYSQL_PS1="Session 1 \R:\m:\s> " mysql -v -A --host=database-1.cluster-ce5fwv4akhjp.eu-central-1.rds.amazonaws.com --port=3306 --user=admin --password=ServerlessV2

I connect a first session , displaying the time and session in the prompt.

Session 1 23:11:55> use demo;
Database changed

Session 1 23:12:15> truncate table t;
--------------
truncate table t
--------------

Query OK, 0 rows affected (0.09 sec)

Session 1 23:12:29> insert into t values ( 1, 1 );
--------------
insert into t values ( 1, 1 )
--------------

Query OK, 1 row affected (0.08 sec)

this just resets the testcase when I want to re-run it.

Session 1 23:12:36> start transaction;
--------------
start transaction
--------------

Query OK, 0 rows affected (0.07 sec)

Session 1 23:12:48> update t set x = 42;
--------------
update t set x = 42
--------------

Query OK, 1 row affected (0.11 sec)
Rows matched: 1  Changed: 1  Warnings: 0

Session 1 23:12:55> rollback;
--------------
rollback
--------------

Query OK, 0 rows affected (0.02 sec)

I updated one row, and rolled back my transaction. This is to show that you must be aware that calling a lambda is out of the ACID protection of relational databases. The trigger is executed during the update, without knowing if the transaction will be committed or not (voluntarily or because an exception is encountered). When you do only things in the database (like writing into another table) there is no problem because this happens within the transaction. If the transaction is rolled back, all the DML done by the triggers are rolled back as well. Even if they occurred, nobody sees their effect, except the current session, before the whole transaction is committed.

But when you call a lambda, synchronously or asynchronously, the call is executed and its effect will not be rolled back if your transaction does not complete. This can be ok in some cases, if what you execute is related to the intention of the update and not its completion. Or you must manage this exception in your lambda, maybe by checking in the database that the transaction occurred. But in that case, you should really question your architecture (a call to a service, calling back to the caller…)

So… be careful with that. If your lambda is there to be executed when a database operation has been done, it may have to be done after the commit, in the procedural code that has executed the transaction.

Another test…

This non-ACID execution was the important point I wanted to emphasize, so you can stop here if you want. This other test is interesting for people used to Oracle Database only, probably. In general, nothing guarantees that a trigger is executed only once for the triggering operation. What we have seen above (rollback) can be done internally when a serialization exception is encountered and the database can retry the operation. Oracle Database has non-blocking reads and this is not only for SELECT but also for the read phase of an UPDATE. You may have to read a lot of rows to verify the predicate and update only a few ones, and you don’t want to lock all the rows read but only the ones that are updated. Manually, you would do that with a serializable transaction and retry in case you encounter a rows that have been modified between your MVCC snapshot and the current update time. But at statement level, Oracle does that for you.

It seems that it does not happen in Aurora MySQL and PostgreSQL, as the locking for reads is more aggressive, but just in case I tested the same scenario where an update restart would have occurred in Oracle.

Session 1 23:13:00> start transaction;
--------------
start transaction
--------------

Query OK, 0 rows affected (0.06 sec)

Session 1 23:13:09> update t set x = x+1;
--------------
update t set x = x+1
--------------

Query OK, 1 row affected (0.02 sec)
Rows matched: 1  Changed: 1  Warnings: 0

Session 1 23:13:16> select * from t;
--------------
select * from t
--------------

+------+------+
| x    | y    |
+------+------+
|    2 |    1 |
+------+------+
1 row in set (0.01 sec)

Session 1 23:13:24>

I have started a transaction that increased the value of X, but the transaction is still open. What I do next is from another session.

This is session 2:

Session 2 23:13:32> use demo;

Database changed
Session 2 23:13:34>
Session 2 23:13:35> select * from t;
--------------
select * from t
--------------

+------+------+
| x    | y    |
+------+------+
|    1 |    1 |
+------+------+
1 row in set (0.01 sec)

Of course, thanks to transaction isolation, I do not see the uncommitted change.

Session 2 23:13:38> update t set x = x+1 where x > 0;
--------------
update t set x = x+1 where x > 0
--------------

At this step, the update hangs on the locked row.

Now back in the first session:

Session 1 23:13:49>
Session 1 23:13:50>
Session 1 23:13:50>
Session 1 23:13:50> commit;
--------------
commit
--------------

Query OK, 0 rows affected (0.02 sec)

I just commited my change here, so X has been increased to the value 2.

And here is what happened in my seconds session, with the lock released by the first session:

Query OK, 1 row affected (11.42 sec)
Rows matched: 1  Changed: 1  Warnings: 0

Session 2 23:13:58> commit;
--------------
commit
--------------

Query OK, 0 rows affected (0.01 sec)

Session 2 23:14:10> select * from t;
--------------
select * from t
--------------

+------+------+
| x    | y    |
+------+------+
|    3 |    2 |
+------+------+
1 row in set (0.01 sec)

Session 2 23:14:18>

This is the correct behavior. Even if a select sees the value of X=1 the update cannot be done until the first session has committed its transaction. This is why it waited, and it has read the committed value of X=2 which is then incremented to 3.

And finally here is what was logged by my lambda, as a screenshot and as text:

2020-12-13T23:12:55.558+01:00	START RequestId: 39e4e41f-7853-4b11-a12d-4a3147be3fc7 Version: $LATEST	2020/12/13/[$LATEST]25e73c8c6f9e4d168fa29b9ad2ba76d8
2020-12-13T23:12:55.561+01:00	Hello.	2020/12/13/[$LATEST]25e73c8c6f9e4d168fa29b9ad2ba76d8
2020-12-13T23:12:55.561+01:00	{'trigger': 't_bufer', 'connection': '124', 'old': '1', 'new': '42'}	2020/12/13/[$LATEST]25e73c8c6f9e4d168fa29b9ad2ba76d8
2020-12-13T23:12:55.562+01:00	END RequestId: 39e4e41f-7853-4b11-a12d-4a3147be3fc7	2020/12/13/[$LATEST]25e73c8c6f9e4d168fa29b9ad2ba76d8
2020-12-13T23:12:55.562+01:00	REPORT RequestId: 39e4e41f-7853-4b11-a12d-4a3147be3fc7 Duration: 1.16 ms Billed Duration: 2 ms Memory Size: 128 MB Max Memory Used: 51 MB	2020/12/13/[$LATEST]25e73c8c6f9e4d168fa29b9ad2ba76d8
2020-12-13T23:13:16.620+01:00	START RequestId: 440128db-d6de-4b2c-aa98-d7bedf12a3d4 Version: $LATEST	2020/12/13/[$LATEST]25e73c8c6f9e4d168fa29b9ad2ba76d8
2020-12-13T23:13:16.624+01:00	Hello.	2020/12/13/[$LATEST]25e73c8c6f9e4d168fa29b9ad2ba76d8
2020-12-13T23:13:16.624+01:00	{'trigger': 't_bufer', 'connection': '124', 'old': '1', 'new': '2'}	2020/12/13/[$LATEST]25e73c8c6f9e4d168fa29b9ad2ba76d8
2020-12-13T23:13:16.624+01:00	END RequestId: 440128db-d6de-4b2c-aa98-d7bedf12a3d4	2020/12/13/[$LATEST]25e73c8c6f9e4d168fa29b9ad2ba76d8
2020-12-13T23:13:16.624+01:00	REPORT RequestId: 440128db-d6de-4b2c-aa98-d7bedf12a3d4 Duration: 1.24 ms Billed Duration: 2 ms Memory Size: 128 MB Max Memory Used: 51 MB	2020/12/13/[$LATEST]25e73c8c6f9e4d168fa29b9ad2ba76d8
2020-12-13T23:13:58.156+01:00	START RequestId: c50ceab7-6e75-4e43-b77d-26c1f6347fec Version: $LATEST	2020/12/13/[$LATEST]25e73c8c6f9e4d168fa29b9ad2ba76d8
2020-12-13T23:13:58.160+01:00	Hello.	2020/12/13/[$LATEST]25e73c8c6f9e4d168fa29b9ad2ba76d8
2020-12-13T23:13:58.160+01:00	{'trigger': 't_bufer', 'connection': '123', 'old': '2', 'new': '3'}	2020/12/13/[$LATEST]25e73c8c6f9e4d168fa29b9ad2ba76d8
2020-12-13T23:13:58.160+01:00	END RequestId: c50ceab7-6e75-4e43-b77d-26c1f6347fec	2020/12/13/[$LATEST]25e73c8c6f9e4d168fa29b9ad2ba76d8
2020-12-13T23:13:58.160+01:00	REPORT RequestId: c50ceab7-6e75-4e43-b77d-26c1f6347fec Duration: 0.91 ms Billed Duration: 1 ms Memory Size: 128 MB Max Memory Used: 51 MB	2020/12/13/[$LATEST]25e73c8c6f9e4d168fa29b9ad2ba76d8

First, we see at 23:12:55 the update from X=1 to X=42 that I rolled back later. This proves that the call to lambda is not transactional. It may sound obvious but if you come from Oracle Database you would have used Advanced Queuing where the queue is stored in a RDBMS table and then benefit from sharing the same transaction as the submitter.
My update occurred at 23:12:48 but remember that those calls are asynchronous so the log happens a bit later.

Then there was my second test where I updated, at 23:13:09, X from 1 to 2 and we see this update logged at 23:13:16 which is after the update, for the asynchronous reason, but before the commit which happened at 23:13:50 according to my session log above. Then no doubt that the execution of the lambda does not wait for the completion of the transaction that triggered it.

And then the update from the session 2 which was executed at 23:13:38 but returned at 23:13:50 as it was waiting for the first session to end its transaction. The lambda log at 23:13:58 shows it and shows that the old value is X=2 which is expected as the update was done after the first session change. This is where, in Oracle, we would have seen two entries: one updating from X=1, because this would have been read without lock, and then rolled back to restart the update after X=2. But we don’t have this problem here as MySQL acquires a row lock during the read phase.

However, nothing guarantees that there are no internal rollback + restart. And anyway, rollback can happen for many reasons and you should think, during design, whether the call to the lambda should occur for DML intention or DML completion. For example, if you use it for some event sourcing, you may accept the asynchronous delay, but you don’t want to receive an event that actually didn’t occur.

L’article Amazon Aurora: calling a lambda from a trigger est apparu en premier sur dbi Blog.

.
This follows my previous post https://www.dbi-services.com/blog/aurora-serverless-v2-ram/ ‎which you should read before this one. I was looking at the auto-scaling of RAM and it is now time to look at the CPU Utilization.

I have created an Aurora Serverless v2 database (please don’t forget it is the beta preview) with auto-scaling from 4 ACU to 32 ACU. I was looking at a table scan to show how the buffer pool is dynamically resized with auto-scaling. Here I’ll start to run this same cpu() procedure in one, then two, then tree… concurrent sessions to show auto-scaling and related metrics.

Here is the global workload in number of queries per second (I have installed PMM on AWS in a previous post so let’s use it):

And the summary of what I’ve run, with the auto-scaled capacity and the CPU utilization measured:

10:38 1 session  running,  6 ACU , 14% CPU usage
10:54 2 sessions running, 11 ACUs, 26% CPU usage
11:09 3 sessions running, 16 ACUs, 39% CPU usage
11:25 4 sessions running, 21 ACUs, 50% CPU usage
11:40 5 sessions running, 26 ACUs, 63% CPU usage
11:56 6 sessions running, 31 ACUs, 75% CPU usage
12:12 7 sessions running, 32 ACUs, 89% CPU usage
12:27 8 sessions running, 32 ACUs, 97% CPU usage

The timestamp shows when I started to add one more session running in CPU, so that we can match with the metrics from CloudWatch. From there, it looks like the Aurora database engine is running on an 8 vCPU machine and the increase of ACU did not change dynamically the OS threads the “CPU Utilization” metric is based on.

Here are the details from CloudWatch:

The metrics are:

• Serverless Capacity Units on top-left: the auto-scaled ACU from 4 to 32 (in the preview), with a granularity of 0.5
• CPU Utilization on top-right: the sessions running in CPU as a pourcentage of available threads
• Engine Uptime on bottom-left: there were no restart during those runs
• DB connections on botton right: I had 4 idle sessions before starting, then substract 4 and you have the sessions running

With 8 sessions in CPU, I’ve saturated the CPU and, as we reached 100%, my guess is that those are 8 cores, not hyperthreaded. As this is 32 ACUs, this would mean that an ACU is 1/4th of a core, but…

Here is the same metric I displayed from PMM, but here from CloudWatch, to look again how the workload scales:

If ACUs were proportional to the OS cores, I would expect linear performance, which is not the case. One session runs at 1.25M queries per second on 6 ACUs. Two sessions are at 1.8M queries per second on 11 ACUs. Tree sessions at 2.5M queries/s on 16 ACU. So the math is not so simple. Does this mean that 16 ACU does not offer the same throughput as two times 8 ACU? Are we on burstable instances for small ACU? And, 8 vCPU with 64 GB, does that mean that when I start a serverless database with a 32 ACU maximum it runs on a db.r5.2xlarge, whatever the actual ACU it scales to? Is the VM simply provisioned on the maximum ACU and CPU limited by cgroup or similar?

I’ve done another test, this time fixing the min and max ACU to 16. So, maybe, this is similar to provisioning a db.r5.xlarge.
And I modified my cpu() procedure to stop after 10 million loops:

delimiter $
drop procedure if exists cpu;
create procedure cpu()
begin
 declare i int default 0;
 while i < 1e7  do
  set i = i + 1;
 end while;
end$
delimiter ;

1 million loops, this takes 50 seconds on dbfiddle, and you can test it on other platforms where you have an idea of the CPU speed.

I’ve run a loop that connects, run this function and displays the time and loop again:

Dec 07 18:41:45 real    0m24.271s
Dec 07 18:42:10 real    0m25.031s
Dec 07 18:42:35 real    0m25.146s
Dec 07 18:43:00 real    0m24.817s
Dec 07 18:43:24 real    0m23.868s
Dec 07 18:43:48 real    0m24.180s
Dec 07 18:44:12 real    0m23.758s
Dec 07 18:44:36 real    0m24.532s
Dec 07 18:45:00 real    0m23.651s
Dec 07 18:45:23 real    0m23.540s
Dec 07 18:45:47 real    0m23.813s
Dec 07 18:46:11 real    0m24.295s
Dec 07 18:46:35 real    0m23.525s

This is one session and CPU usage is 26% here (this is why I think that my 16 ACU serverless database runs on a 4 vCPU server)

Dec 07 18:46:59 real    0m24.013s
Dec 07 18:47:23 real    0m24.318s
Dec 07 18:47:47 real    0m23.845s
Dec 07 18:48:11 real    0m24.066s
Dec 07 18:48:35 real    0m23.903s
Dec 07 18:49:00 real    0m24.842s
Dec 07 18:49:24 real    0m24.173s
Dec 07 18:49:49 real    0m24.557s
Dec 07 18:50:13 real    0m24.684s
Dec 07 18:50:38 real    0m24.860s
Dec 07 18:51:03 real    0m24.988s

This is two sessions (I’m displaying the time for one only) and CPU usage is 50% which confirms my guess: I’m using half of the CPU resources. And the response time per session is till the same as when one session only was running.

Dec 07 18:51:28 real    0m24.714s
Dec 07 18:51:53 real    0m24.802s
Dec 07 18:52:18 real    0m24.936s
Dec 07 18:52:42 real    0m24.371s
Dec 07 18:53:06 real    0m24.161s
Dec 07 18:53:31 real    0m24.543s
Dec 07 18:53:55 real    0m24.316s
Dec 07 18:54:20 real    0m25.183s

I am now running 3 sessions there and the response time is still similar (I am at 75% CPU usage so obviously I have more than 2 cores here – no hyperthreading – or I should have seen some performance penalty when running more threads than cores)

Dec 07 18:54:46 real    0m25.937s
Dec 07 18:55:11 real    0m25.063s
Dec 07 18:55:36 real    0m24.400s
Dec 07 18:56:01 real    0m25.223s
Dec 07 18:56:27 real    0m25.791s
Dec 07 18:57:17 real    0m24.798s
Dec 07 18:57:42 real    0m25.385s
Dec 07 18:58:07 real    0m24.561s

This was with 4 sessions in total. The CPU is near 100% busy and the response time is still ok, which confirms I have 4 cores available to run that.

Dec 07 18:58:36 real    0m28.562s
Dec 07 18:59:06 real    0m30.618s
Dec 07 18:59:36 real    0m30.002s
Dec 07 19:00:07 real    0m30.921s
Dec 07 19:00:39 real    0m31.931s
Dec 07 19:01:11 real    0m32.233s
Dec 07 19:01:43 real    0m32.138s
Dec 07 19:02:13 real    0m29.676s
Dec 07 19:02:44 real    0m30.483s

One more session here. Now the CPU is a 100% and the processes have to wait 1/5th of their time in runqueue as there is only 4 threads available. That’s an additional 20% that we can see in the response time.

Not starting more processes, but increasing the capacity now, setting the maximum ACU to 24 which then enables auto-scaling:

...
Dec 07 19:08:02 real    0m33.176s
Dec 07 19:08:34 real    0m32.346s
Dec 07 19:09:01 real    0m26.912s
Dec 07 19:09:25 real    0m24.319s
Dec 07 19:09:35 real    0m10.174s
Dec 07 19:09:37 real    0m1.704s
Dec 07 19:09:39 real    0m1.952s
Dec 07 19:09:41 real    0m1.600s
Dec 07 19:09:42 real    0m1.487s
Dec 07 19:10:07 real    0m24.453s
Dec 07 19:10:32 real    0m25.794s
Dec 07 19:10:57 real    0m24.917s
...
Dec 07 19:19:48 real    0m25.939s
Dec 07 19:20:13 real    0m25.716s
Dec 07 19:20:40 real    0m26.589s
Dec 07 19:21:06 real    0m26.341s
Dec 07 19:21:34 real    0m27.255s

At 19:00 I increased to maximum ACU to 24 and let it auto-scale. The engine restarted at 19:09:30 and I got some errors until 19:21 where I reached the optimal response time again. I have 5 sessions running on a machine sized for 24 ACUs which I think is 6 OS threads and then I expect 5/6=83% CPU utilization if all my hypothesis are right. Here are the CloudWatch metrics:

Yes, it seems we reached this 83% after some fluctuations. Those irregularities may be the consequence of my scripts running loops of long procedures. When the engine restarted (visible in “Engine Uptime”), I was disconnected for a while (visible in “DB Connections”), then the load decreased (visible in “CPU Utilization”), then scaling-down the available resources (visible in “Serverless Capacity Unit”)

The correspondence between ACU and RAM is documented (visible when defining the min/max and reported in my previous post) and the the instance types for provisioned Aurora gives the correspondance between RAM and vCPU (which confirms what I’ve seen here 16 ACU 32GB 4 vCPU as a db.r5.xlarge): https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/Concepts.DBInstanceClass.html#aurora-db-instance-classes

Please remember, all those are guesses as very little information is disclosed about how it works internally. And this is a preview beta, many things will be different when GA. The goal of this blog is only to show that a little understanding about how it works will be useful when deciding between provisioned or serverless, think about side effects, and interpret the CloudWatch metrics. And we don’t need huge workloads for this investigation: learn on small labs and validate it on real stuff.

L’article Aurora Serverless v2 (preview) – CPU est apparu en premier sur dbi Blog.

.
What is Aurora Serverless? That’s the RDS Aurora name for auto-scaling: instead of provisioning an instance size (from the burstable db.t3.small with 2 vCPU and 2GB RAM to db.r5.16xlarge with 64 vCPU and 512 GB RAM) you define a range in term of ACU /Aurora Capacity Unit). ACU is about CPU + RAM. This blog post will focus on RAM.

Aurora Serverless v1

In Serverless v1 the ACU goes from 1 (2 GB RAM) to 256 (488GB RAM) and the granularity is in power of two: each scale-up will double the instance. You can also opt for a minimum capacity of 0 where the instance is stopped when unused (no connections for 5 minutes), but then you accept that it takes few minutes for the first connection after that to startup again to the minimum capacity defined. Scaling happens on measured metrics like CPU (scale-up when >70%, down when <30%). The number of connections (percentage of max), and available RAM (but this is actually how the maximum number of maximum connections is calculated from the instance RAM. I don’t think the RAM threshold considers the usage of the shared buffer pool in serverless v1. Aurora tries to scale-up as soon as one threshold is reached, but scale-down requires both CPU and connections to be lower than the threshold, and scale-down cannot happen within 15 minutes after scale-up (the cooldown period). And, as scaling in Serverless v1 means stopping the database and starting it in another VM, it tries to do it outside of active connections and may timeout or force (your choice).

Serverless 1 will still be available. And there are currently some features that are not available in v2, like PostgreSQL compatibility or Data API. But they will come and I suppose v2 will replace v1 one day.

Aurora Serverless v2 (preview)

This new service has a finer grain of auto-scaling. With server’s virtualization, there’s the possibility to increase the number of vCPU on a VM without restart, and MySQL 5.7.5 can change the buffer pool online. And this gives a finer grain in scaling up and scaling down (announced 0.5 ACU gain), and without waiting. The preview goes from 4 ACU (8GB) to 32 (64GB) but the plan is that the minimum is as low as 0.5 ACU and up to 256 ACU. Then, you will probably not opt for stopping the instance, to avoid cold start latency, but keep it low at 0.5 ACU and then the database will be immediately available when a connection comes. And the granularity is by addition of 0.5 ACU rather than doubling the instance. So, even if the ACU is more expensive in v2, you probably consume less. And the scale-down doesn’t have to wait 15 minutes to cut by half the capacity as it can be decreased progressively online. Of course, having the instance restarted is still a possibility if there’s no vCPU available in the shared one, but that should not happen often.

Here is an example where I created a 8GB demo table:

--------------
create procedure demo(n int)
begin
 declare i int default 0;
 create table demo (id int not null primary key auto_increment, n int,x varchar(1000));
 insert into demo(n,x) values (1,lpad('x',898,'x'));
 while i < n do
  set i = i + 1;
  insert into demo(n,x) select n,x from demo;
  commit;
 end while;
end
--------------

--------------
call demo(23)
--------------

VM movement

During this creation I experienced multiple scale down (after the instance creation) and up (during the table row ingestion) and you can see that in this case the VM was probably moved to another server and had to restart. The “Engine Uptime” in CloudWatch testifies from the restarts and “Serverless Database Capacity” i the ACU (capacity units):

During those VM movements, I got this kind of error:

ERROR 2013 (HY000) at line 29: Lost connection to MySQL server during query

You just have to retry in this case. If you can’t then you will set some minimum/maximum ACU or maybe go to a provisioned database.

--------------
analyze table demo
--------------
Table           Op       Msg_type   Msg_text
-----------     -------  --------   --------
franck.demo     analyze  status     OK

--------------
select round(data_length/1024/1024) TABLE_MB,round(index_length/1024/1024) INDEX_MB,table_rows,table_name,table_type,engine from information_s
chema.tables where table_schema='franck'
--------------

TABLE_MB  INDEX_MB        table_rows      table_name      table_type      engine
--------  --------        ----------      ----------      ----------      ------
    8200         0           7831566            demo      BASE TABLE      InnoDB

Here I checked the size of my table: about 8GB.

buffer pool

I mentioned vCPU but what about RAM? The VM memory can also be resized online but there’s a difference. With CPU, if you scaled-down too early, it can scale-up immediately and you get back to the previous performance. But when you do that with RAM you have evicted some data from the caches that will not be back immediately until the first sessions warms it up again. So, the Serverless v2 has to look at the InnoDB LRU (Least Recently Used) buffers to estimate the risk to drop them. I mention InnoDB because for the moment Aurora Serverless v2 is on the MySQL compatibility only.

On my DEMO table I’ve run the following continuously:

use franck;
set profiling = 1;
select count(*) from demo where x='$(date) $RANDOM';
show profiles;

I’ve run that in a loop, so one session continuously active reading 8GB (the predicate does not filter anything and is there just to run a different query each time as I want to show the effect on the buffer pool and not the query cache).

Then, from 18:00 to 18:23 approximately I have run another session:

use franck;
delimiter $$
drop procedure if exists cpu;
create procedure cpu()
begin
 declare i int default 0;
 while 1  do
  set i = i + 1;
 end while;
end$$
delimiter ;
call cpu();

Please, don’t judge me on my MySQL procedural code I’m just looping in CPU
Then after 20 minutes:

MySQL [(none)]> show full processlist;

--------------
show full processlist
--------------

+-----+----------+--------------------+--------+---------+------+--------------+------------------------------------------------------------------------+
| Id  | User     | Host               | db     | Command | Time | State        | Info                                                                   |
+-----+----------+--------------------+--------+---------+------+--------------+------------------------------------------------------------------------+
|  42 | admin    | 192.169.29.1:22749 | franck | Query   |    0 | NULL         | call cpu()                                                             |
| 302 | rdsadmin | localhost          | NULL   | Sleep   |    1 | NULL         | NULL                                                                   |
| 303 | rdsadmin | localhost          | NULL   | Sleep   |    0 | NULL         | NULL                                                                   |
| 304 | rdsadmin | localhost          | NULL   | Sleep   |    0 | NULL         | NULL                                                                   |
| 305 | rdsadmin | localhost          | NULL   | Sleep   |    0 | NULL         | NULL                                                                   |
| 339 | admin    | 192.169.29.1:30495 | NULL   | Query   |    0 | starting     | show full processlist                                                  |
| 342 | admin    | 192.169.29.1:42711 | franck | Query   |    3 | Sending data | select count(*) from demo where x='Sun Dec  6 18:23:25 CET 2020 28911' |
+-----+----------+--------------------+--------+---------+------+--------------+------------------------------------------------------------------------+
7 rows in set (0.00 sec)

MySQL [(none)]> kill 42;
--------------
kill 42
--------------

I’ve stopped my running loop.
And here is what I can see from CloudWatch:

• DB Connection: I always have one busy connection most of the time for the repeated scans on DEMO. And during 20 minutes a second one (my CPU procedure). The third one at the end is when I connected to kill the session
• Serverless Database Capacity: this is the number of ACU. The value was 6 when only the scan was running, and scaled-up to 11 when the CPU session was running
• CPU Utilization (Percent): this is the percentage on the number of OS threads. 13.6% when only the scan session was running, and reached 23% during the additional CPU run. Note that the number of OS threads was not increased when ACU scaled from 6 to 11… I’ll come back on that later
• Engine Uptime: increasing by 1 minute every minute, this means that all scale up/down was done without a restart of the engine

Now, something that you don’t see in CloudWatch metrics, is the response time for my DEMO table scan.

Dec 06 17:52:37 1       79.03579950     select count(*) from demo where x='Sun Dec  6 17:51:17 CET 2020 13275'
Dec 06 17:53:59 1       79.83154300     select count(*) from demo where x='Sun Dec  6 17:52:37 CET 2020 4418'
Dec 06 17:55:20 1       79.81895825     select count(*) from demo where x='Sun Dec  6 17:53:59 CET 2020 11596'
Dec 06 17:56:40 1       78.29040100     select count(*) from demo where x='Sun Dec  6 17:55:20 CET 2020 25484'
Dec 06 17:58:02 1       80.15728125     select count(*) from demo where x='Sun Dec  6 17:56:40 CET 2020 15321'
Dec 06 17:59:42 1       98.29309550     select count(*) from demo where x='Sun Dec  6 17:58:02 CET 2020 31126'
Dec 06 18:01:09 1       85.07732725     select count(*) from demo where x='Sun Dec  6 17:59:42 CET 2020 29792'
Dec 06 18:02:30 1       79.16154650     select count(*) from demo where x='Sun Dec  6 18:01:09 CET 2020 21930'
Dec 06 18:02:34 1       2.81377450      select count(*) from demo where x='Sun Dec  6 18:02:30 CET 2020 12269'
Dec 06 18:02:38 1       2.77996150      select count(*) from demo where x='Sun Dec  6 18:02:34 CET 2020 30306'
Dec 06 18:02:42 1       2.73756325      select count(*) from demo where x='Sun Dec  6 18:02:38 CET 2020 22678'
Dec 06 18:02:47 1       2.77504400      select count(*) from demo where x='Sun Dec  6 18:02:42 CET 2020 933'
Dec 06 18:02:51 1       2.73966275      select count(*) from demo where x='Sun Dec  6 18:02:47 CET 2020 21922'
Dec 06 18:02:56 1       2.87023975      select count(*) from demo where x='Sun Dec  6 18:02:51 CET 2020 9158'
Dec 06 18:03:00 1       2.75959675      select count(*) from demo where x='Sun Dec  6 18:02:56 CET 2020 31710'
Dec 06 18:03:04 1       2.72658975      select count(*) from demo where x='Sun Dec  6 18:03:00 CET 2020 27248'
Dec 06 18:03:09 1       2.71731325      select count(*) from demo where x='Sun Dec  6 18:03:04 CET 2020 18965'

More than one minute to scan those 8GB, that’s 100 MB/s which is what we can expect from physical reads. 8GB doesn’t fit in a 6 ACU instance. Then, when I started another session, which triggered auto-scaling to 11 ACU, the memory became large enough and this is why my response time for the scan is now less than 3 seconds. I mentioned that I’d come back on CPU usage because that’s not easy to do the maths without looking at the OS. I think that this will deserve another blog post. I have seen 13.6% of CPU Utilization when the count was running alone in CPU. There were I/O involved here, but as far as I know 13.6% on 6 ACU is the equivalent of one sessions running in CPU. So probably the scan was not throttled by I/O? Then I added another session, which I know was running fully in CPU, and the scan was also running fully in CPU (all from buffer pool) between the connection time. I had 23% CPU Utilization and I think that in a 11 ACU scale, 2 sessions fully on CPU take 26%. I’ll do other test to try to confirm this. I miss Performance Insight here to get the real picture…

Then, as you have seen that scale-down happened when I stopped my concurrent session, you can imagine the response time:

Dec 06 18:24:23 1       3.03963650      select count(*) from demo where x='Sun Dec  6 18:24:17 CET 2020 2815'
Dec 06 18:24:27 1       2.80417800      select count(*) from demo where x='Sun Dec  6 18:24:23 CET 2020 16763'
Dec 06 18:24:31 1       2.77208025      select count(*) from demo where x='Sun Dec  6 18:24:27 CET 2020 29473'
Dec 06 18:24:36 1       3.13085700      select count(*) from demo where x='Sun Dec  6 18:24:31 CET 2020 712'
Dec 06 18:24:41 1       2.77904025      select count(*) from demo where x='Sun Dec  6 18:24:36 CET 2020 17967'
Dec 06 18:24:45 1       2.76111900      select count(*) from demo where x='Sun Dec  6 18:24:41 CET 2020 20407'
Dec 06 18:24:49 1       2.79092475      select count(*) from demo where x='Sun Dec  6 18:24:45 CET 2020 7644'
Dec 06 18:26:17 1       85.68287300     select count(*) from demo where x='Sun Dec  6 18:24:49 CET 2020 691'
Dec 06 18:27:40 1       81.58135400     select count(*) from demo where x='Sun Dec  6 18:26:17 CET 2020 14101'
Dec 06 18:29:02 1       80.00523900     select count(*) from demo where x='Sun Dec  6 18:27:40 CET 2020 31646'
Dec 06 18:30:22 1       78.79213700     select count(*) from demo where x='Sun Dec  6 18:29:02 CET 2020 811'
Dec 06 18:31:42 1       78.37765950     select count(*) from demo where x='Sun Dec  6 18:30:22 CET 2020 24539'
Dec 06 18:33:02 1       78.64492525     select count(*) from demo where x='Sun Dec  6 18:31:42 CET 2020 789'
Dec 06 18:34:22 1       78.36776750     select count(*) from demo where x='Sun Dec  6 18:33:02 CET 2020 2321'
Dec 06 18:35:42 1       78.38105625     select count(*) from demo where x='Sun Dec  6 18:34:22 CET 2020 27716'
Dec 06 18:37:04 1       79.74060525     select count(*) from demo where x='Sun Dec  6 18:35:42 CET 2020 487'

Yes, down to 6 ACU, the buffer pool is shrinked, back to physical I/O…

This is the risk when CPU and RAM are scaled in proportion: a single-thread may not have enough RAM in order to save on CPU. And, paradoxically, can get more when there are more concurrent activity. Here is the kind of scenario I would not like to encounter:

Yes. That, and estimating the cost of warming-up the cache when the scale-up needs to move the VM and restart the engine. Didn't test it but I foresee: higher load -> scale up -> move VM -> more I/O -> less CPU% (until cache warmed up) –> a scale down here would be a bad idea.

— Franck Pachot (@FranckPachot) December 6, 2020

ACU and Buffer Pool size

I’ve seen 0 ACU only during the creation, but in this preview there is no “pause” option and we can go only between 4 and 32 ACU.

I have tested all of them to check the related settings. You see ACU and max_connections in x-axis, and VM size as well as the buffer size and query cache size:

Here you see what happened with my 8GB demo table, it didn’t fit in a 6 ACU shape where the buffer pool is 6GB but stayed in memory in the 11 ACU shape with 13.5 GB.
Among the interesting buffer pool settings that are different in Serverless:

--------------
show global variables like '%innodb%buffer_pool%'
--------------

Variable_name   Value
innodb_buffer_pool_chunk_size   157286400
innodb_buffer_pool_dump_at_shutdown     OFF
innodb_buffer_pool_dump_now     OFF
innodb_buffer_pool_dump_pct     25
innodb_buffer_pool_filename     ib_buffer_pool
innodb_buffer_pool_instances    8
innodb_buffer_pool_load_abort   OFF
innodb_buffer_pool_load_at_startup      OFF
innodb_buffer_pool_load_now     OFF
innodb_buffer_pool_size 3774873600
innodb_shared_buffer_pool_uses_huge_pages       OFF
--------------

Huge pages are off in Serverless. This innodb_shared_buffer_pool_uses_huge_pages is not a MySQL but an Aurora specific one which is ON with the provisioned flavor of Aurora, but off in serverless. This makes sense given how Serverless can allocate and de-allocate memory.

ACU and CPU

As I mentioned, this deserves a new blog post. Amazon does not give the ACU – vCPU equivalence. And, given the CPU Utilization percentages I see, I don’t think that the VM is resized. Except when we see that the engine is restarted.

The price

About the price, I let you read Jeremy Daly analysis: https://www.jeremydaly.com/aurora-serverless-v2-preview/
My opinion… serverless is a feature that the cloud provider gives you to lower their revenue. Then, of course, it has to be more expensive. The cloud provider must keep a margin of idle CPU in order to face your scale-out without moving the VM (taking more time and flushing memory, which compromises availability). You pay more when busy, but you save on idle time without risking saturation at peak.

And anyway, don’t forget that Serverless is an option. It may fit your needs or not. If you don’t want the buffer pool effect that I’ve described above, you can provision an instance where you know exactly how much RAM you have. And… don’t forget this is preview, like beta, and anything can change. Next post gives more details about CPU Utilization: https://www.dbi-services.com/blog/aurora-serverless-v2-cpu/

L’article Aurora Serverless v2 (preview) – RAM est apparu en premier sur dbi Blog.

.
This year is not very nice for conferences as everything is virtual and we miss the most important: meeting and sharing with people. But the AWS re:Invent is actually a great experience. As an AWS Data Heros, I received an Oculus Quest 2 to teleport to the virtual Neon City where we can meet and have fun in Virtual Reality (but incredibly real-life chatting):

A little bit of what @AWSreInvent looks like this year for the #AWSHeroes in #VirtualReality! Amazing experience from @awscloud!

What are your thoughts in this experience for an online event?#aws #awsreinvent2020 #vr #awsreinvent pic.twitter.com/Lhz51iLcyw

— Filipe Barretto (@filipe_barretto) November 30, 2020

There are 3 important new launches announced around databases: Babelfish for Aurora, Aurora Serverless v2 and AWS Glue Elastic Views but let’s start by a recap of the pre-reInvent new features from this year.

We have more regions, even one planned in Switzerland. And also more cloud at customer solutions, like RDS in Outposts in addition to RDS on VMware. We had new versions, PostgreSQL 12, MariaDB 10.5, SQL Server 2019. SQL Server even came with SSRS). And also recent Release Updates for Oracle (July 2020).
About new features from 2020, we can export RDS snapshots to S3 parquet format. We can share AD with RDS from multiple VPC, we have connection pooling in RDS Proxy (session state aware). SQL Server supports parallel backups. Oracle supports backup to other regions. RDS can use always-on for SQL Server read replicas. And Oracle does not need Active Data Guard option when the replica is not there for read workloads. And talking about licenses, there’s the License Manager for Oracle to help manage them. There’s also the new Graviton2 processors for RDS PostgreSQL and MySQL.

All that was about relational databases, there’s also new features in NoSQL databases, like DynamoDB export to S3, PartiQL queries. But let’s new go to the new launches.

AWS Glue Elastic Views

I mentioned that we can query the NoSQL DynamoDB tables with a SQL-like API, PartiQL. Now those PartiQL queries can do more: continuous query to propagate data and changes, like materialized views. This event sourcing is based on CDC (not Stream). It propagates changes in near real-time (asynchronous, can be throttled by the target capacity) and to multiple destinations: Elasticsearch to search, S3 for data lake, Redshift for analytics. A nice serverless solution for CQRS: DynamoDB for ingest and OLTP and propagation to purpose-build services for the queries that cannot be done in the NoSQL operational database. This is serverless: billed per second of compute, and volume of storage.

Currently, those materialized views support only selection and projection, but hopefully, in the future, they will be able to maintain aggregations with GROUP BY. As I’m not a fan of writing procedural code to process data, I really like materialized views for replication, rather than triggers and lambdas.

Aurora Serverless v2

You don’t want to pre-plan the capacity but have your database server scale up, out, and down according to the load? That’s serverless. You don’t provision servers, but capacity units: Aurora Capacity Units (ACU). Rather than multiplying the capacity when needed, by changing the instance size, the new Aurora Serverless v2 elasticity has a granularity of 0.5 ACU: you start by provisioning 0.5 ACU (not zero because you don’t want to wait seconds on first start after being idle). When compared with v1 (which is still available) the starting capacity is lower, the increment is finer, and the scale-down is in minute rather than a 15 minutes cool down. And it has all Aurora features: Multi-AZ, Global Database, Proxy,… Basically, this relies on the ability to add vCPU and memory online, and reduce it (this includes shrinking the buffer pool according to LRU). This means scale up and down as long as it is possible (depends on the neighbors activity in the same VM). It can scale out as well if in the compute fleet and move to another VM if needed, but the goal is to be able to scale-up in-place most of the time.

Releasing idle CPU is easy, but knowing how much RAM can be released without significantly increase I/O and response time, is probably more challenging. Anyway, we can expect min/max controls on it. The goal is not to replace the capacity planning, but to be more elastic with unplanned workloads.

You have the choice to migrate to v2, but look at the price. The ACU is more expensive, but given the elasticity, you probably save a lot (start lower, increase by smaller steps, decrease sooner).

Babelfish

This is the most revolutionary in my opinion. We want polyglot databases not only to have the coice of language or API for new developments. Many databases run applications, like ERP, which are tied to a specific commercial database. And companies want to get out of this vendor lock-in but migration of those applications is impossible. They use specific behaviour, or code, in the database, and they do it for a reason: the agility and performance of processing data within the database. The business logic is tied to data for consistency and performance, in stored procedures. There are many attempts to translate the code, but this works partially. And that’s not sufficient for enterprise software: rewriting is easy but testing… who will sign the UAT validation that the business code, working for years in a database engine, has been rewritten to show the same behaviour?

This is different when there is no application change at all, and that’s the idea of Babelfish, starting with SQL Server compatibility in Aurora. Given the powerful extensibility of PostgreSQL, AWS has built some extensions to understand T-SQL, and specific SQL Server datatype behaviour. They also add endpoints that understand the MS SQL network protocol. And then can run the applications running on SQL Server, without any change besides the connection to the new endpoint. Of course, this is not easy. Each application may have specificities and need to implement new extensions. And for this reason, AWS decided to Open Source this compatibility layer. Who will contribute? Look at the ISV who has an ERP running on SQL Server. They can invest in developing the compatibility with Babelfish, and then can propose to their customer to move out of the commercial database, to PostgreSQL. Of course, the goal of AWS is to get them to Aurora, providing the high availability and scalability that big companies may require. But Babelfish target is PostgreSQL, the community one.

About the target, Aurora comes with two flavors, using the upper layer from MySQL or PostgreSQL. PostgreSQL was chosen as it is probably the most compatible with commercial databases, and provides easy extensibility in procedural language, datatypes and extensions. About the source, it is SQL Server for the moment (a commercial reply to the licensing policy they have set for their cloud competitors) but I’m sure Oracle will come one day. Probably not 100% compatible, given the complexity of it, but the goal of an ISV is to provide 100% compatibility for one application. And, once compatibility is there, the database is also accessible with the native PostgreSQL API for further developments.

I’m looking forward to seeing how this Open Source project will get contributions. Aurora has a bad reputation in the PostgreSQL community, taking the community code, making money with it, and not giving back their optimizations. But this Babelfish can really extend the popularity of this reliable open-source database. Contributions are not only extensions for code-compatibility. I can expect lot of contributions about test cases and documentation.

I’ve seen a demo about T-SQL and Money datatype. This is nice, but a single-user test case. I’ll test concurrency as soon as I have the preview. Isolation of transactions, read and write consistency in multi-user workloads are very different in PostgreSQL and SQL Server. And test case for compatibility acceptance is not easy.

You can expect more technical insights on this blog, as soon as I have access to the preview. For the moment, let me share some pictures about the Oculus Quest 2 I got from the AWS Heroes program, and the Neon City place where we meet:

I forgot to mention the io2 Block Express which will be very interesting for database bandwidth with 4GB/s (and 256K IOPS if you really need this):

AWS io2 Block Express is multi-attach. Like a SAN in the Cloud. Will be tempting to try Oracle RAC
Oh… did I forget that production use of Oracle RAC on
"Third-Party Clouds" is not permitted by this list: https://t.co/KYrofX1Z1E https://t.co/Hb6O24wXd4

— Franck Pachot (@FranckPachot) December 1, 2020

and the EC2 R5b instance:
https://twitter.com/ClossonAtWork/status/1334300318104834054?s=20

L’article Database announcements at re:Invent 2020 est apparu en premier sur dbi Blog.

.
In the previous post I described the PartiSQL SELECT for DynamoDB and mentioned that a SELECT without a WHERE clause on the partition key may result in a Scan, but the result is automatically paginated. This pagination, and the cost of a Scan, is something that may not be very clear from the documentation and I’ll show it here on the regular DynamoDB API. By not very clear, I think this is why many people in the AWS community fear that, with this new PartiQL API, there is a risk to full scan tables, consuming expensive RCUs. I was also misled, when I started to look at DynamoDB, by the AWS CLI “–no-paginate” option, as well as its “Consumed Capacity” always showing 128.5 even for very large scans. So those examples should, hopefully, clear out some doubts.

I have created a HASH/RANGE partitioned table and filled it with a few thousands of items:

aws dynamodb create-table --attribute-definitions \
  AttributeName=MyKeyPart,AttributeType=N \
  AttributeName=MyKeySort,AttributeType=N \
 --key-schema \
  AttributeName=MyKeyPart,KeyType=HASH \
  AttributeName=MyKeySort,KeyType=RANGE \
 --billing-mode PROVISIONED \
 --provisioned-throughput ReadCapacityUnits=25,WriteCapacityUnits=25 \
 --table-name Demo

for i in {1..5000} ; do aws dynamodb put-item --table-name Demo --item '{"MyKeyPart":{"N":"'$(( $RANDOM /1000 ))'"},"MyKeySort":{"N":"'$SECONDS'"},"MyUnstructuredData":{"S":"'$(printf %-1000s | tr ' ' x)'"}}' ; done

Here is how those items look like:

I have created large items with a 1000 bytes “MyUnstructuredData” attribute. According to https://zaccharles.github.io/dynamodb-calculator/ an item size is around 1042 bytes. And that’s exactly the size I see here (5209783/5000=1041.96) from the console “Items summary” (I waited a few hours to get it updated in the screenshot above). This means that around 1000 items can fit on a 1MB page. We will see why I’m mentioning 1MB here: the title says 128.5 RCU and that’s the consumed capacity when reading 1MB with eventual consistency (0.5 RCU per 4KB read is 128 RCU per 1MB). Basically, this post will try to explain why we see a 128.5 consumed capacity at maximum when scanning any large table:

[opc@a aws]$ aws dynamodb scan --table-name Demo --select=COUNT --return-consumed-capacity TOTAL\
 --no-consistent-read --output table

----------------------------------
|              Scan              |
+----------+---------------------+
|   Count  |    ScannedCount     |
+----------+---------------------+
|  5000    |  5000               |
+----------+---------------------+
||       ConsumedCapacity       ||
|+----------------+-------------+|
||  CapacityUnits |  TableName  ||
|+----------------+-------------+|
||  128.5         |  Demo       ||
|+----------------+-------------+|
[opc@a aws]$

TL;DR: this number is wrong
You cannot scan 5000 items of 1000 bytes with 128.5 RCU as this is nearly 5MB scanned and you need 0.5 RCU per 4KB reads.

Output text

I’ve run this with “–output table” for a pretty print of it. Let’s have a look at the other formats (json and text):

[opc@a aws]$ aws dynamodb scan --table-name Demo --select=COUNT --return-consumed-capacity TOTAL \
--no-consistent-read --output json
{
    "Count": 5000,
    "ScannedCount": 5000,
    "ConsumedCapacity": {
        "TableName": "Demo",
        "CapacityUnits": 128.5
    }
}

[opc@a aws]$ aws dynamodb scan --table-name Demo --select=COUNT --return-consumed-capacity TOTAL\
 --no-consistent-read --output text

1007    None    1007
CONSUMEDCAPACITY        128.5   Demo
1007    None    1007
1007    None    1007
1007    None    1007
972     None    972

The JSON format is similar to the TABLE one, but the TEXT output gives more information about this 128.5 consumed capacity as it appears after a count of 1007 items. Yes, this makes sense, 1007 is approximately the number of my items I expect in 1MB and, as I mentioned earlier, reading 1MB in eventual consistency consumes 128 RCU (0.5 RCU per 4KB). What actually happens here is pagination. A scan call is always limited to read 1MB at maximum (you can compare that to a fetch size in a SQL database except that it is about the amount read rather than returned) and what happens here is that the AWS CLI fetches the next pages in order to get the whole COUNT. Unfortunately, the “–return-consumed-capacity TOTAL” shows the value from the first fetch only. And only the TEXT format shows this count for each call. The TABLE and JSON formats do the sum of count for you (which is nice) but hide the fact that the Consumed Capacity is the one from the first call only.

Debug

The partial display of the consumed capacity is a problem with AWS CLI but each call actually returns the right value, which we can see with “–debug”:

[opc@a aws]$ aws dynamodb scan --table-name Demo --select=COUNT --return-consumed-capacity TOTAL \
 --no-consistent-read --output table --debug 2>&1 | grep '"ConsumedCapacity"'

b'{"ConsumedCapacity":{"CapacityUnits":128.5,"TableName":"Demo"},"Count":1007,"LastEvaluatedKey":{"MyKeyPart":{"N":"2"},"MyKeySort":{"N":"96744"}},"ScannedCount":1007}'
b'{"ConsumedCapacity":{"CapacityUnits":128.5,"TableName":"Demo"},"Count":1007,"LastEvaluatedKey":{"MyKeyPart":{"N":"27"},"MyKeySort":{"N":"91951"}},"ScannedCount":1007}'
b'{"ConsumedCapacity":{"CapacityUnits":128.5,"TableName":"Demo"},"Count":1007,"LastEvaluatedKey":{"MyKeyPart":{"N":"20"},"MyKeySort":{"N":"85531"}},"ScannedCount":1007}'
b'{"ConsumedCapacity":{"CapacityUnits":128.5,"TableName":"Demo"},"Count":1007,"LastEvaluatedKey":{"MyKeyPart":{"N":"29"},"MyKeySort":{"N":"90844"}},"ScannedCount":1007}'
b'{"ConsumedCapacity":{"CapacityUnits":124.0,"TableName":"Demo"},"Count":972,"ScannedCount":972}'

Here the total RCU is 128.5+128.5+128.5+128.5+125=639 which is what we can expect here to scan a 5MB table (0.5*5209783/4096=636).

Pagination

To make the confusion bigger, there are several meanings in “pagination”. One is about the fact that a scan call reads at maximum 1MB of DynamoDB storage (and then at maximum 128.5 RCU – or 257 for strong consistency). And the other is about the fact that, from the AWS CLI, we can automatically fetch the next pages.

I can explicitly ask to read only one page with the “–no-paginate” option (misleading name, isn’t it? It actually means “do not automatically read the next pages”):

[opc@a aws]$ aws dynamodb scan --table-name Demo --select=COUNT --return-consumed-capacity TOTAL \
--no-consistent-read --output json --no-paginate

{
    "Count": 1007,
    "ScannedCount": 1007,
    "LastEvaluatedKey": {
        "MyKeyPart": {
            "N": "2"
        },
        "MyKeySort": {
            "N": "96744"
        }
    },
    "ConsumedCapacity": {
        "TableName": "Demo",
        "CapacityUnits": 128.5
    }
}

[opc@a aws]$ aws dynamodb scan --table-name Demo --select=COUNT --return-consumed-capacity TOTAL \
--no-consistent-read --output text --no-paginate

1007    1007
CONSUMEDCAPACITY        128.5   Demo
MYKEYPART       2
MYKEYSORT       96744

Here, in all output formats, things are clear as the consumed capacity matches the number of items. The first 1MB page has 1007 items, which consumes 128.5 RCU, and, in order to know the total number, we need to read the next pages.

Different than the auto pagination (automatically call the next page until the end), the read pagination always happens for scans: you will never read more than 1MB from the DynamoDB storage in one call. This is how the API avoids any surprise in the response time: the fetch size depends on the cost of data access rather than the result. If I add a filter to my scan so that no rows are returned, the same pagination happens:

[opc@a aws]$ aws dynamodb scan --table-name Demo --select=COUNT --return-consumed-capacity TOTAL \
--no-consistent-read --output table \
--filter-expression "MyUnstructuredData=:v" --expression-attribute-values '{":v":{"S":"franck"}}'

----------------------------------
|              Scan              |
+----------+---------------------+
|   Count  |    ScannedCount     |
+----------+---------------------+
|  0       |  5000               |
+----------+---------------------+
||       ConsumedCapacity       ||
|+----------------+-------------+|
||  CapacityUnits |  TableName  ||
|+----------------+-------------+|
||  128.5         |  Demo       ||
|+----------------+-------------+|

[opc@a aws]$ aws dynamodb scan --table-name Demo --select=COUNT --return-consumed-capacity TOTAL \
--no-consistent-read --output text \
--filter-expression "MyUnstructuredData=:v" --expression-attribute-values '{":v":{"S":"franck"}}'

0       None    1007
CONSUMEDCAPACITY        128.5   Demo
0       None    1007
0       None    1007
0       None    1007
0       None    972

Here no items verify my filter (Count=0) but all items had to be scanned (ScanndCount=5000). And then, as displayed with the text output, pagination happened, returning empty pages. This is a very important point to understand DynamoDB scans: as there is no access filter (no key value) it does a Full Table Scan, with the cost of it, and the filtering is done afterwards. This means that empty pages can be returned and you may need multiple roundtrips even for no rows. And this is what I had here with the COUNT: 5 calls to get the answer “Count: 0”.

For my Oracle Database readers, you can think of DynamoDB scan operation like a “TABLE ACCESS FULL” in an execution plan (but not like a “TABLE ACCESS STORAGE FULL” which offloads the predicates to the storage) where you pay per throttled reads per second. The cost of the operation depends on the volume read (the size of the table) but not on the result. Yes, the message in DynamoDB is “avoid scan as much as possible” like we had “avoid full table scans as much as possible” in SQL databases, when used for OLTP, except for small tables. And guess what? The advantage of DynamoDB scan operation comes when you need to read a large part of the table because it can read many items with one call, with 1MB read I/O size on the storage… Yes, 1MB, the same as what the db_file_multiblock_read_count default value has always set for maximum I/O size behind in Oracle for full table scans. APIs change but many concepts are the same.

Page size

We can control the page size, if we want more smaller pages (but you need very good reasons to do so), by specifying the number of items. In order to show that it is about the number of items scanned but not returned after filtering, I keep my filter that removes all items from the result:

[opc@a aws]$ aws dynamodb scan --table-name Demo --select=COUNT --return-consumed-capacity TOTAL \
--no-consistent-read --output text --filter-expression "MyUnstructuredData=:v" --expression-attribute-values '{":v":{"S":"franck"}}' \
--page-size 500

0       None    500
CONSUMEDCAPACITY        64.0    Demo
0       None    500
0       None    500
0       None    500
0       None    500
0       None    500
0       None    500
0       None    500
0       None    500
0       None    500
0       None    0

Here each page is smaller than before, limited to 500 items. Then the RCU consumed by each call is smaller, as well as the response time. However, it is clear that the total number of RCU consumed is still the same, even higher:

[opc@a aws]$ aws dynamodb scan --table-name Demo --select=COUNT --return-consumed-capacity TOTAL \
--no-consistent-read --output text --filter-expression "MyUnstructuredData=:v" --expression-attribute-values '{":v":{"S":"franck"}}' \
--page-size 10 \
--debug  2>&1 | awk -F, '/b.{"ConsumedCapacity":{"CapacityUnits":/{sub(/[^0-9.]*/,"");cu=cu+$1}END{print cu}'

750.5

and the response time is higher because of additional roundtrips.

We can define smaller pages, but never larger than 1MB:

[opc@a aws]$ aws dynamodb scan --table-name Demo --select=COUNT --return-consumed-capacity TOTAL \
--no-consistent-read --output text --filter-expression "MyUnstructuredData=:v" --expression-attribute-values '{":v":{"S":"franck"}}' \
--page-size 1500

0       None    1067
CONSUMEDCAPACITY        128.5   Demo
0       None    1074
0       None    1057
0       None    1054
0       None    748

Here, even if I asked for 1500 items per page, I get the same as before because, given the item size, the 1MB limit is reached first.

A few additional tests:

[opc@a aws]$ aws dynamodb scan --table-name Demo --select=COUNT --return-consumed-capacity TOTAL \
--no-consistent-read --output table --page-size 1500 --no-paginate                                                                                             

Cannot specify --no-paginate along with pagination arguments: --page-size

[opc@a aws]$ aws dynamodb scan --table-name Demo --select=COUNT --return-consumed-capacity TOTAL \
--no-consistent-read --output table --page-size 1 --debug 2>&1 | grep '"ConsumedCapacity"' | head -2

b'{"ConsumedCapacity":{"CapacityUnits":0.5,"TableName":"Demo"},"Count":1,"LastEvaluatedKey":{"MyKeyPart":{"N":"7"},"MyKeySort":{"N":"84545"}},"ScannedCount":1}'
b'{"ConsumedCapacity":{"CapacityUnits":0.5,"TableName":"Demo"},"Count":1,"LastEvaluatedKey":{"MyKeyPart":{"N":"7"},"MyKeySort":{"N":"85034"}},"ScannedCount":1}'

[opc@a aws]$ aws dynamodb scan --table-name Demo --select=COUNT --return-consumed-capacity TOTAL \
--no-consistent-read --output table --page-size 10 --debug 2>&1 | grep '"ConsumedCapacity"' | head -2

b'{"ConsumedCapacity":{"CapacityUnits":1.5,"TableName":"Demo"},"Count":10,"LastEvaluatedKey":{"MyKeyPart":{"N":"7"},"MyKeySort":{"N":"85795"}},"ScannedCount":10}'
b'{"ConsumedCapacity":{"CapacityUnits":1.5,"TableName":"Demo"},"Count":10,"LastEvaluatedKey":{"MyKeyPart":{"N":"7"},"MyKeySort":{"N":"86666"}},"ScannedCount":10}'

[opc@a aws]$ aws dynamodb scan --table-name Demo --select=COUNT --return-consumed-capacity TOTAL \
--no-consistent-read --output table --page-size 100 --debug 2>&1 | grep '"ConsumedCapacity"' | head -2

b'{"ConsumedCapacity":{"CapacityUnits":13.0,"TableName":"Demo"},"Count":100,"LastEvaluatedKey":{"MyKeyPart":{"N":"7"},"MyKeySort":{"N":"94607"}},"ScannedCount":100}'
b'{"ConsumedCapacity":{"CapacityUnits":13.0,"TableName":"Demo"},"Count":100,"LastEvaluatedKey":{"MyKeyPart":{"N":"8"},"MyKeySort":{"N":"89008"}},"ScannedCount":100}'

First, I cannot disable auto pagination when defining a page size. This is why I used the debug mode to get the RCU consumed. Reading only one item (about 1KB) consumed 0.5 RCU because this is the minimum: 0.5 to read up to 4KB. Then I called for 10 and 100 items per page. This helps to estimate the size of items. 13 RCU for 100 items means that the average item size is 13*4096/0.5/100=1065 bytes.

You probably don’t want to reduce the page size under 1MB, except maybe if your RCU are throttled and you experience timeout. It is a response time vs. throughput decision. And in any case, a scan page should return many items. If I scan all my 5000 items with –page-size 1 will require 2500 RCU because each call is 0.5 at minimum:

[opc@a aws]$ aws dynamodb scan --table-name Demo --select=COUNT --return-consumed-capacity TOTAL --no-consistent-read --output text --filter-expression "MyUnstructuredData=:v" --expression-attribute-values '{":v":{"S":"franck"}}' --page-size 1 --debug  2>&1 | awk -F, '/b.{"ConsumedCapacity":{"CapacityUnits":/{sub(/[^0-9.]*/,"");cu=cu+$1}END{print cu}'

2500.5

This cost with the smallest page size is the same as reading each item with a getItem operation. So you see, except with this extremely small page example, that scan is not always evil. When you need to read many items, scan can get them with less RCU. How much is “many”? The maths is easy here. With 0.5 RCU you can read a whole 1MB page with scan, or just one item with getItem. Then, as long as, on average, you read more than one item per page, you get a benefit from scan. You can estimate the number of items you retreive. And you can divide the table size by 1MB. But keep in mind that if the table grows, the cost of scan increases. And sometimes, you prefer scalable and predictable response time over fast response time.

Max items

In addition to the page size (number of items scanned) we can also paginate the result. This doesn’t work for count:

[opc@a aws]$ aws dynamodb scan --table-name Demo --select=COUNT --return-consumed-capacity TOTAL \
--no-consistent-read --output text --max-items 1

1007    None    1007
CONSUMEDCAPACITY        128.5   Demo
1007    None    1007
1007    None    1007
1007    None    1007
972     None    972

Here, despite the “–max-items 1” the full count has been returned.

I’m now selecting (projection) two attributes, with a “–max-items 5”:

[opc@a aws]$ aws dynamodb scan --table-name Demo --select=SPECIFIC_ATTRIBUTES --projection-expression=MyKeyPart,MyKeySort \
--return-consumed-capacity TOTAL --no-consistent-read --output text --max-items 5

1007    1007
CONSUMEDCAPACITY        128.5   Demo
MYKEYPART       7
MYKEYSORT       84545
MYKEYPART       7
MYKEYSORT       85034
MYKEYPART       7
MYKEYSORT       85182
MYKEYPART       7
MYKEYSORT       85209
MYKEYPART       7
MYKEYSORT       85359
NEXTTOKEN       eyJFeGNsdXNpdmVTdGFydEtleSI6IG51bGwsICJib3RvX3RydW5jYXRlX2Ftb3VudCI6IDV9

This, like pagination, gives a “next token” to get the remaining items.

[opc@a aws]$ aws dynamodb scan --table-name Demo --select=SPECIFIC_ATTRIBUTES --projection-expression=MyKeyPart,MyKeySort \
--return-consumed-capacity TOTAL --no-consistent-read --output text --max-items 5 \
--starting-token eyJFeGNsdXNpdmVTdGFydEtleSI6IG51bGwsICJib3RvX3RydW5jYXRlX2Ftb3VudCI6IDV9

0       0
CONSUMEDCAPACITY        128.5   Demo
MYKEYPART       7
MYKEYSORT       85380
MYKEYPART       7
MYKEYSORT       85516
MYKEYPART       7
MYKEYSORT       85747
MYKEYPART       7
MYKEYSORT       85769
MYKEYPART       7
MYKEYSORT       85795
NEXTTOKEN       eyJFeGNsdXNpdmVTdGFydEtleSI6IG51bGwsICJib3RvX3RydW5jYXRlX2Ftb3VudCI6IDEwfQ==

The displayed cost here is the same as a full 1MB scan: 128.5 RCU and, if you look at the calls with “–debug” you will see that a thousand of items were returned. However, the ScannedCount is zero:

[opc@a aws]$ aws dynamodb scan --table-name Demo --select=SPECIFIC_ATTRIBUTES --projection-expression=MyKeyPart,MyKeySort \
--return-consumed-capacity TOTAL --no-consistent-read --output table --max-items 5 \
--starting-token eyJFeGNsdXNpdmVTdGFydEtleSI6IG51bGwsICJib3RvX3RydW5jYXRlX2Ftb3VudCI6IDV9

-----------------------------------------------------------------------------------------------------------
|                                                  Scan                                                   |
+-------+---------------------------------------------------------------------------------+---------------+
| Count |                                    NextToken                                    | ScannedCount  |
+-------+---------------------------------------------------------------------------------+---------------+
|  0    |  eyJFeGNsdXNpdmVTdGFydEtleSI6IG51bGwsICJib3RvX3RydW5jYXRlX2Ftb3VudCI6IDEwfQ==   |  0            |
+-------+---------------------------------------------------------------------------------+---------------+
||                                           ConsumedCapacity                                            ||
|+---------------------------------------------------------+---------------------------------------------+|
||                      CapacityUnits                      |                  TableName                  ||
|+---------------------------------------------------------+---------------------------------------------+|
||  128.5                                                  |  Demo                                       ||
|+---------------------------------------------------------+---------------------------------------------+|
||                                                 Items                                                 ||

Does it make sense? No items scanned but thousand if items retrieved, with the RCU of a 1MB read?

Let’s try to answer this. I’ll query this again and update one item:

[opc@a aws]$ aws dynamodb scan --table-name Demo --select=ALL_ATTRIBUTES --return-consumed-capacity TOTAL <
--no-consistent-read --output text --max-items 5 \
--starting-token eyJFeGNsdXNpdmVTdGFydEtleSI6IG51bGwsICJib3RvX3RydW5jYXRlX2Ftb3VudCI6IDV9 \
| cut -c1-80

0       0
CONSUMEDCAPACITY        128.5   Demo
MYKEYPART       7
MYKEYSORT       85380
MYUNSTRUCTUREDDATA      xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
MYKEYPART       7
MYKEYSORT       85516
MYUNSTRUCTUREDDATA      xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
MYKEYPART       7
MYKEYSORT       85747
MYUNSTRUCTUREDDATA      xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
MYKEYPART       7
MYKEYSORT       85769
MYUNSTRUCTUREDDATA      xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
MYKEYPART       7
MYKEYSORT       85795
MYUNSTRUCTUREDDATA      xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
NEXTTOKEN       eyJFeGNsdXNpdmVTdGFydEtleSI6IG51bGwsICJib3RvX3RydW5jYXRlX2Ftb3VudCI6ID

[opc@a aws]$ aws dynamodb execute-statement \
--statement "update Demo set MyunstructuredData='Hello' where MyKeyPart=7 and MyKeySort =85516"

------------------
|ExecuteStatement|
+----------------+

I’ve used the PartiQL SQL-Like API juszt because I find it really convenient for this.

Now scanning from the same next token:

[opc@a aws]$ aws dynamodb scan --table-name Demo --select=ALL_ATTRIBUTES --return-consumed-capacity TOTAL <
--no-consistent-read --output text --max-items 5 \
--starting-token eyJFeGNsdXNpdmVTdGFydEtleSI6IG51bGwsICJib3RvX3RydW5jYXRlX2Ftb3VudCI6IDV9 \
| cut -c1-80

0       0
CONSUMEDCAPACITY        128.5   Demo
MYKEYPART       7
MYKEYSORT       85380
MYUNSTRUCTUREDDATA      xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
MYKEYPART       7
MYKEYSORT       85516
MYUNSTRUCTUREDDATA      xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
MYUNSTRUCTUREDDATA      Hello
MYKEYPART       7
MYKEYSORT       85747
MYUNSTRUCTUREDDATA      xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
MYKEYPART       7
MYKEYSORT       85769
MYUNSTRUCTUREDDATA      xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
MYKEYPART       7
MYKEYSORT       85795
MYUNSTRUCTUREDDATA      xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
NEXTTOKEN       eyJFeGNsdXNpdmVTdGFydEtleSI6IG51bGwsICJib3RvX3RydW5jYXRlX2Ftb3VudCI6ID

Besides the fact that my update has added a new attribute MyunstructuredData rather than replacing MyUnstructuredData because I made a typo (not easy to spot with the text output as it uppercases all attributes names) the important point is that I’ve read the new value. Despite the “ScannedCount=0”, I have obviously read the items again. Nothing stays in cache and there are no stateful cursors in a NoSQL database.

So just be careful with “–max-items”. It limits the result, but not the work done in one page read. RCU is always calculated from the number of 4KB that are read to get the page from the storage, far before any filtering. Where “–max-items” can limit the cost is when using auto pagination to avoid reading more pages than necessary:

[opc@a aws]$ aws dynamodb scan --table-name Demo --select=SPECIFIC_ATTRIBUTES --projection-expression=MyKeyPart,MyKeySort --return-consumed-capacity TOTAL \
--no-consistent-read --output text \
--max-items 2500 --debug 2>&1 | grep '"ConsumedCapacity"' | cut -c1-80

b'{"ConsumedCapacity":{"CapacityUnits":128.5,"TableName":"Demo"},"Count":1007,"I
b'{"ConsumedCapacity":{"CapacityUnits":128.5,"TableName":"Demo"},"Count":1007,"I
b'{"ConsumedCapacity":{"CapacityUnits":128.5,"TableName":"Demo"},"Count":1007,"I

Here limiting the displayed result to 2500 items only 3 pages (of 1007 items which are not filtered to the result) have been read.

Consistency

I’ve run all those scans with “–no-consistent-read”, which is the default, just to make it implicit that we accept to miss the latest changes. With consistent reads, we are sure to read the latest, but requires more reads (from a quorum of mirrors) and doubles the RCU consumption:

[opc@a aws]$ aws dynamodb scan --table-name Demo --select=SPECIFIC_ATTRIBUTES --projection-expression=MyKeyPart,MyKeySort \
--return-consumed-capacity TOTAL --consistent-read --output text --max-items 5 \
--starting-token eyJFeGNsdXNpdmVTdGFydEtleSI6IG51bGwsICJib3RvX3RydW5jYXRlX2Ftb3VudCI6IDV9

0       0
CONSUMEDCAPACITY        257.0   Demo
MYKEYPART       7
MYKEYSORT       85380
MYKEYPART       7
MYKEYSORT       85516
MYKEYPART       7
MYKEYSORT       85747
MYKEYPART       7
MYKEYSORT       85769
MYKEYPART       7
MYKEYSORT       85795
NEXTTOKEN       eyJFeGNsdXNpdmVTdGFydEtleSI6IG51bGwsICJib3RvX3RydW5jYXRlX2Ftb3VudCI6ID==

The cost here is 257 RCU as 4KB consistent reads cost 1 RCU instead of 0.5 without caring about consistency.

Query

I focused on the scan operation here, but the same multi-item read applies to the query operation. Except that you do not read the whole table, even with pagination, as you define a specific partition by providing a value for the partition key which will be hashed to one partition.

Those tests are fully reproducible on the Free Tier. You don’t need billions of items to understand how it works. And once you understand how it works, simple math will tell you how it scales to huge tables.

L’article DynamoDB Scan (and why 128.5 RCU?) est apparu en premier sur dbi Blog.

.
I recently installed Percona Monitoring & Management on AWS (free tier) and here is how to monitor an instance on another cloud (OCI), in order to show which TCP port must be opened.

PMM server

I installed PMM from the AWS Marketplace, following those instructions: https://www.percona.com/doc/percona-monitoring-and-management/deploy/server/ami.html. I’ll not reproduce the instructions, just some screenshots I took during the install:

I have opened the HTTPS port in order to access the console, and also configure the clients which will also connect through HTTPS (but I’m not using a signed certificate).

Once installed, two targets are visible: the PMM server host (Linux) and database (PostgreSQL):

Note that I didn’t secure HTTPS here and I’ll have to accept insecure SSL.

PMM client

I’ll monitor an Autonomous Linux instance that I have on Oracle Cloud (Free Tier). Autonomous Linux is based on OEL, which is based on RHEL (see https://www.dbi-services.com/blog/al7/) and is called “autonomous” because it updates the kernel without the need to reboot. I install the PMM Client RPM:

[opc@al ~]$ sudo yum -y install https://repo.percona.com/yum/percona-release-latest.noarch.rpm


Loaded plugins: langpacks
percona-release-latest.noarch.rpm                                                                        |  19 kB  00:00:00
Examining /var/tmp/yum-root-YgvokG/percona-release-latest.noarch.rpm: percona-release-1.0-25.noarch
Marking /var/tmp/yum-root-YgvokG/percona-release-latest.noarch.rpm to be installed
Resolving Dependencies
--> Running transaction check
---> Package percona-release.noarch 0:1.0-25 will be installed
--> Finished Dependency Resolution
al7/x86_64                                                                                               | 2.8 kB  00:00:00
al7/x86_64/primary_db                                                                                    |  21 MB  00:00:00
epel-apache-maven/7Server/x86_64                                                                         | 3.3 kB  00:00:00
ol7_UEKR5/x86_64                                                                                         | 2.5 kB  00:00:00
ol7_latest/x86_64                                                                                        | 2.7 kB  00:00:00
ol7_x86_64_userspace_ksplice                                                                             | 2.8 kB  00:00:00

Dependencies Resolved

Dependencies Resolved

================================================================================================================================
 Package                        Arch                  Version               Repository                                     Size
================================================================================================================================
Installing:
 percona-release                noarch                1.0-25                /percona-release-latest.noarch                 31 k

Transaction Summary
================================================================================================================================
Install  1 Package

Total size: 31 k
Installed size: 31 k
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded

Running transaction
  Installing : percona-release-1.0-25.noarch                                                                                1/1
* Enabling the Percona Original repository
 All done!
* Enabling the Percona Release repository
 All done!
The percona-release package now contains a percona-release script that can enable additional repositories for our newer products.

For example, to enable the Percona Server 8.0 repository use:

  percona-release setup ps80

Note: To avoid conflicts with older product versions, the percona-release setup command may disable our original repository for some products.

For more information, please visit:
  https://www.percona.com/doc/percona-repo-config/percona-release.html

  Verifying  : percona-release-1.0-25.noarch                                                                                1/1

Installed:
  percona-release.noarch 0:1.0-25

This packages helps to enable additional repositories. Here, I need the PMM 2 Client:

[opc@al ~]$ sudo percona-release enable pmm2-client


* Enabling the PMM2 Client repository
 All done!

Once enabled, it is easy to install it with YUM:

[opc@al ~]$ sudo yum install -y pmm2-client


Loaded plugins: langpacks
percona-release-noarch                                                                                   | 2.9 kB  00:00:00
percona-release-x86_64                                                                                   | 2.9 kB  00:00:00
pmm2-client-release-x86_64                                                                               | 2.9 kB  00:00:00
prel-release-noarch                                                                                      | 2.9 kB  00:00:00
(1/4): percona-release-noarch/7Server/primary_db                                                         |  24 kB  00:00:00
(2/4): pmm2-client-release-x86_64/7Server/primary_db                                                     | 3.5 kB  00:00:00
(3/4): prel-release-noarch/7Server/primary_db                                                            | 2.5 kB  00:00:00
(4/4): percona-release-x86_64/7Server/primary_db                                                         | 1.2 MB  00:00:00
Resolving Dependencies
--> Running transaction check
---> Package pmm2-client.x86_64 0:2.11.1-6.el7 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

================================================================================================================================
 Package                     Arch                   Version                        Repository                              Size
================================================================================================================================
Installing:
 pmm2-client                 x86_64                 2.11.1-6.el7                   percona-release-x86_64                  42 M

Transaction Summary
================================================================================================================================
Install  1 Package

Total download size: 42 M
Installed size: 42 M
Downloading packages:
warning: /var/cache/yum/x86_64/7Server/percona-release-x86_64/packages/pmm2-client-2.11.1-6.el7.x86_64.rpm: Header V4 RSA/SHA256
 Signature, key ID 8507efa5: NOKEY
Public key for pmm2-client-2.11.1-6.el7.x86_64.rpm is not installed
pmm2-client-2.11.1-6.el7.x86_64.rpm                                                                      |  42 MB  00:00:07
Retrieving key from file:///etc/pki/rpm-gpg/PERCONA-PACKAGING-KEY
Importing GPG key 0x8507EFA5:
 Userid     : "Percona MySQL Development Team (Packaging key) "
 Fingerprint: 4d1b b29d 63d9 8e42 2b21 13b1 9334 a25f 8507 efa5
 Package    : percona-release-1.0-25.noarch (@/percona-release-latest.noarch)
 From       : /etc/pki/rpm-gpg/PERCONA-PACKAGING-KEY
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : pmm2-client-2.11.1-6.el7.x86_64                                                                              1/1
  Verifying  : pmm2-client-2.11.1-6.el7.x86_64                                                                              1/1

Installed:
  pmm2-client.x86_64 0:2.11.1-6.el7

Complete!

That’s all for software installation. I just need to configure the agent to connect to the PMM Server:

[opc@al ~]$ sudo pmm-admin config --server-url https://admin:secretpassword@18.194.119.174 --server-insecure-tls $(curl ident.me) generic OPC-$(hostname)


Checking local pmm-agent status...
pmm-agent is running.
Registering pmm-agent on PMM Server...
Registered.
Configuration file /usr/local/percona/pmm2/config/pmm-agent.yaml updated.
Reloading pmm-agent configuration...
Configuration reloaded.
Checking local pmm-agent status...
pmm-agent is running.

As you can see, I use the “ident-me” web service to identify my IP address, but you probably know your public IP.

This configuration goes to a file, which you should protect because it contains the password in clear text:

[opc@al ~]$ ls -l /usr/local/percona/pmm2/config/pmm-agent.yaml
-rw-r-----. 1 pmm-agent pmm-agent 805 Nov 28 20:22 /usr/local/percona/pmm2/config/pmm-agent.yaml

# Updated by `pmm-agent setup`.
---
id: /agent_id/853027e6-563e-42b8-a417-f144541358ff
listen-port: 7777
server:
  address: 18.194.119.174:443
  username: admin
  password: secretpassword
  insecure-tls: true
paths:
  exporters_base: /usr/local/percona/pmm2/exporters
  node_exporter: /usr/local/percona/pmm2/exporters/node_exporter
  mysqld_exporter: /usr/local/percona/pmm2/exporters/mysqld_exporter
  mongodb_exporter: /usr/local/percona/pmm2/exporters/mongodb_exporter
  postgres_exporter: /usr/local/percona/pmm2/exporters/postgres_exporter
  proxysql_exporter: /usr/local/percona/pmm2/exporters/proxysql_exporter
  rds_exporter: /usr/local/percona/pmm2/exporters/rds_exporter
  tempdir: /tmp
  pt_summary: /usr/local/percona/pmm2/tools/pt-summary
ports:
  min: 42000
  max: 51999
debug: false
trace: false

What is interesting here is the port that is used for the server to connect to pull metrics from the client: 42000

I’ll need to open this port. I can see the error from the PMM server: https://18.194.119.174/prometheus/targets

I open this port on the host:

[opc@al ~]$ sudo iptables -I INPUT 5 -i ens3 -p tcp --dport 42000 -m state --state NEW,ESTABLISHED -j ACCEPT

and on the ingress rules as well:

Testing

I’m running two processes here to test if I get the right metrics

[opc@al ~]$ while true ; do sudo dd bs=100M count=1                if=$(df -Th | sort -rhk3 | awk '/^[/]dev/{print $1;exit}') of=/dev/null ; done &
[opc@al ~]$ while true ; do sudo dd bs=100M count=10G iflag=direct if=$(df -Th | sort -rhk3 | awk '/^[/]dev/{print $1;exit}') of=/dev/null ; done &

The latter will do mostly I/O as I read with O_DIRECT and the former mainly system CPU as it reads from filesystem cache

Here is the Grafana dashboard from PMM:

I see my two processes, and 80% of CPU stolen by the hypervisor as I’m running on the Free Tier here which provides 1/8th of OCPU.

If you have MySQL or PostgreSQL databases there, they can easily be monitored (“pmm-admin add MySQL” or “pmm-admin add MySQL” you can see all that in Elisa Usai demo: https://youtu.be/VgOR_GCUpVw?t=1558).

Last test, let’s see what happens if the monitored host reboots:

[opc@al ~]$ date
Sat Nov 28 22:54:36 CET 2020
[opc@al ~]$ uptrack-uname -a
Linux al 4.14.35-2025.402.2.1.el7uek.x86_64 #2 SMP Fri Oct 23 22:27:16 PDT 2020 x86_64 x86_64 x86_64 GNU/Linux
[opc@al ~]$ uname -a
Linux al 4.14.35-1902.301.1.el7uek.x86_64 #2 SMP Tue Mar 31 16:50:32 PDT 2020 x86_64 x86_64 x86_64 GNU/Linux
[opc@al ~]$ sudo systemctl reboot
Connection to 130.61.159.88 closed by remote host.
Connection to 130.61.159.88 closed.

Yes… I do not reboot it frequently because it is Autonomous Linux and the Effective kernel is up to date (latest patches from October) even if the last restart was in March. But this deserves a test.

The first interesting thing is that PMM seems to keep the last read metrics for a while:

The host was shut down at 22:55 and it shows the last metrics for 5 minutes before stopping.

I had to wait for a while because my Availability Domain was out of capacity for the free tier:

Sometimes, even an Autonomous Linux must be rebooted. And if you are unlucky, there are no available servers in the Availability Domain and it remains stopped
(this is Free Tier) pic.twitter.com/sd3KIeoHkd

— Franck Pachot (@FranckPachot) November 28, 2020

[opc@al ~]$ systemctl status pmm-agent.service
● pmm-agent.service - pmm-agent
   Loaded: loaded (/usr/lib/systemd/system/pmm-agent.service; enabled; vendor preset: disabled)
   Active: active (running) since Sat 2020-11-28 23:40:10 UTC; 1min 15s ago
 Main PID: 46446 (pmm-agent)
   CGroup: /system.slice/pmm-agent.service
           ├─46446 /usr/sbin/pmm-agent --config-file=/usr/local/percona/pmm2/config/pmm-agent.yaml
           └─46453 /usr/local/percona/pmm2/exporters/node_exporter --collector.bonding --collector.buddyinfo --collector.cpu ...

No problem, the installation of PMM client has defined the agent to restart on reboot.

In summary, PMM pulls the metric from the exporter, so you need to open inbound ports on the host where the PMM client agent runs. And HTTPS on the PMM server. Then everything is straightforward.

L’article Cross-cloud PMM: which TCP ports to open est apparu en premier sur dbi Blog.