One of our customers needed to archive a 4T Oracle database. Several options have been chosen, one of them being archiving the data to SIARD format using SIARD Suite from Swiss Federal Archives. This blog posts explains the basics: what SIARD Suite is, how it works, and examples in command line mode.

Meeting SIARD Suite

SIARD Suite is software developed and made freely available by Swiss Federal Archives for long-term archiving of data stored in the following relational database management systems:

• MS Access 2007 or newer
• DB/2 or newer
• MySQL (or MariaDB) 5.5 or newer
• Oracle 10 or newer
• PostgreSQL 11 or newer
• SQL Server 2012 or newer

Data is extracted and stored in a custom SIARD format based on XML, SQL:2008, UNICODE and ZIP64 standards, thus enabling long-term storage of data archives (often required by audit projects, governance and legal frameworks).

Archive data files can later on be used to load the data in a new database if required.

The software can be downloaded from GitHub through this link.

Concepts

Archived data is stored in an XML files collection conforming to SQL:2008 standard.

Code and custom application objects are not extracted. However, table and columns definition are exported as XML metadata. BLOBs and CLOBs are stored as binary files outside of the XML files.

To unload more metadata, database features can be used such as Data Pump or DBMS_METADATA package procedures/functions when dealing with an Oracle database as the source database.

Basic principles of database archiving with SIARD Suite

The following list summarizes what to take into account and anticipate before starting a new archiving project:

• clearly define the data archiving scope: which tables, views or schemas, which records, and so on
• create a dedicated user with the least privileges (typically read-only) on the objects belonging to the archiving scope
• create views in case you need to filter data (columns and/or rows) to be archived
• the source database must be frozen during archiving jobs (use whatever features the source database provides to bring the database in such a state)
• to check thoroughly the end result of the archiving jobs, load the archive data into another database
• define additional metadata to archive based on your specific business needs ; this will later (think of several years in the future) lead to a better understanding of data archives : PL/SQL code, application documentation, data model entity-relation diagrams, and so on.

How SIARD Suite works

SIARD Suite is a JavaFX application, requiring JRE17 with JavaFX support in its latest release (v2.2.167) and can be used on Windows, Linux and MacOS.

Based on our experience, it is recommended to download the binary file with the right Java version embedded (always better not having to manage a separate stack).

There are two interfaces available:

• Graphical User Interface (GUI)
• Command Line Interface (CLI)

Take care, if you are planning to use the CLI, to not use “native” binaries because they don’t include the CLI.

Application installation

Binary files are downloaded as ZIP files and can be installed anywhere on your system. For our customer project, we decided to create a separate Virtual Machine with adequate storage space thanks to the high performance link between the two VMs.

Once SIARD Suite has been unzipped, the directory looks like:

SIARD-Suite.2.2.161-linux
├── bin
├── conf
├── legal
├── lib
└── release

As expected, the bin directory contains all relevant utilities to perform the archiving or loading jobs:

SIARD-Suite.2.2.161-linux/bin
├── java
├── jfr
├── jrunscript
├── keytool
├── siard-from-db
├── siard-from-db.bat
├── SIARD-Suite
├── SIARD-Suite.bat
├── siard-to-db
└── siard-to-db.bat

Command line interface

‘siard-from-db’ utility

Using the ‘siard-from-db’ utility is quite straightforward since there is no fine grained configurations/options/parameters to choose from:

siard-from-db [-h] | [-o] [-v] [-l=<login timeout>] [-q=<query timeout>] [-i=<import
meta data>] [-x=<external lob folder>] [-m=<mime type>] -j=<JDBC URL> -u=<database
user> -p=<database password> -s=<siard file> -e=<export meta data>"

The following table, pulled from the SIARD Suite documentation, details each parameter’s goal and usage pattern:

For instance, to extract the Oracle HR sample schema, we could use the following command:

./siard-from-db -l=10 -q=120 \
-j=jdbc:oracle:thin:@[hostname|ip]:listening_port/pdb_name \
-u=hr -p=hr_password -s=hr.siard -e=hr_metadata.xml

The archive job produces the following two files:

Running the Linux ‘file’ command on the ‘hr.siard’ file confirms it is a Zip archive file:

file hr.siard
hr.siard: Zip archive data, at least v2.0 to extract

Listing the content of the archive file returns:

unzip -l hr.siard
Archive:  hr.siard
  Length      Date    Time    Name
---------  ---------- -----   ----
        0  07-27-2026 08:15   content/
        0  07-27-2026 08:16   content/schema0/
        0  07-27-2026 08:16   content/schema0/table0/
        0  07-27-2026 08:16   content/schema0/table1/
        0  07-27-2026 08:16   content/schema0/table2/
        0  07-27-2026 08:16   content/schema0/table3/
        0  07-27-2026 08:16   content/schema0/table4/
        0  07-27-2026 08:16   content/schema0/table5/
        0  07-27-2026 08:16   content/schema0/table6/
     1595  07-27-2026 08:16   content/schema0/table0/table0.xml
     4857  07-27-2026 08:16   content/schema0/table0/table0.xsd
     1884  07-27-2026 08:16   content/schema0/table1/table1.xml
     4910  07-27-2026 08:16   content/schema0/table1/table1.xsd
    19577  07-27-2026 08:16   content/schema0/table2/table2.xml
     5323  07-27-2026 08:16   content/schema0/table2/table2.xsd
     1786  07-27-2026 08:16   content/schema0/table3/table3.xml
     4909  07-27-2026 08:16   content/schema0/table3/table3.xsd
     1345  07-27-2026 08:16   content/schema0/table4/table4.xml
     4951  07-27-2026 08:16   content/schema0/table4/table4.xsd
     2740  07-27-2026 08:16   content/schema0/table5/table5.xml
     5036  07-27-2026 08:16   content/schema0/table5/table5.xsd
      455  07-27-2026 08:16   content/schema0/table6/table6.xml
     4793  07-27-2026 08:16   content/schema0/table6/table6.xsd
        0  07-27-2026 08:16   header/
        0  07-27-2026 08:16   header/siardversion/
        0  07-27-2026 08:16   header/siardversion/2.2/
    38441  07-27-2026 08:16   header/metadata.xsd
     4825  07-27-2026 08:16   header/table.xsd
    32980  07-27-2026 08:16   header/metadata.xml
---------                     -------
   140407                     29 files

For each table, SIARD Suite generates an XML file containing the data unloaded from the database and an XSD file with the XML schema definition used.

The header/metadata.xml file contains the list of schemas, list of tables, list of table columns, and so on.

Here is an extract from header/metadata.xml file produced by the HR archiving job:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<siardArchive xmlns="http://www.bar.admin.ch/xmlns/siard/2/metadata.xsd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" version="2.2" xsi:schemaLocation="http://www.bar.admin.ch/xmlns/siard/2/metadata.xsd metadata.xsd">
    <dbname>(...)</dbname>
    <dataOwner>(...)</dataOwner>
    <dataOriginTimespan>(...)</dataOriginTimespan>
    <producerApplication>SiardFromDb 2.2.26 Swiss Federal Archives, Berne, Switzerland, 2008-2016</producerApplication>
    <archivalDate>2026-07-27Z</archivalDate>
    <messageDigest>
        <digestType>MD5</digestType>
        <digest>8DC757A5338AA000234CC6199FEFE0AA</digest>
    </messageDigest>
    <clientMachine>srvsiard</clientMachine>
    <databaseProduct>Oracle Oracle AI Database 26ai Enterprise Edition Release 23.26.2.0.0 - Production</databaseProduct>
    <connection>jdbc:oracle:thin:@xxx.xxx.xxx.xxx:yyyy/pdb_name</connection>
    <databaseUser>HR</databaseUser>
    <schemas>
        <schema>
            <name>HR</name>
            <folder>schema0</folder>
            <tables>
                <table>
                    <name>COUNTRIES</name>
                    <folder>table0</folder>
                    <description>country table. References with locations table.</description>
                    <columns>
                        <column>
                            <name>COUNTRY_ID</name>
                            <type>CHAR(2)</type>
                            <typeOriginal>CHAR(2)</typeOriginal>
                            <nullable>false</nullable>
                            <description>Primary key of countries table.</description>
                        </column>
                        <column>
                            <name>COUNTRY_NAME</name>
                            <type>VARCHAR(60)</type>
                            <typeOriginal>VARCHAR2(60)</typeOriginal>
                            <description>Country name</description>
                        </column>
                        <column>
                            <name>REGION_ID</name>
                            <type>FLOAT(38)</type>
                            <typeOriginal>NUMBER</typeOriginal>
                            <description>Region ID for the country. Foreign key to region_id column in the departments table.</description>
                        </column>
                    </columns>
                    <primaryKey>
                        <name>COUNTRY_C_ID_PK</name>
                        <column>COUNTRY_ID</column>
                    </primaryKey>
                    <foreignKeys>
                        <foreignKey>
                            <name>COUNTR_REG_FK</name>
                            <referencedSchema>HR</referencedSchema>
                            <referencedTable>REGIONS</referencedTable>
                            <reference>
<column>REGION_ID</column>
<referenced>REGION_ID</referenced>
                            </reference>
                            <deleteAction>RESTRICT</deleteAction>
                            <updateAction>CASCADE</updateAction>
                        </foreignKey>
                    </foreignKeys>
                    <rows>25</rows>
                </table>
                <table>
                    <name>DEPARTMENTS</name>
                    <folder>table1</folder>
                    <description>Departments table that shows details of departments where employees\u000Awork. references with locations, employees, and job_history tables.</description>
                    <columns>
                        <column>
                            <name>DEPARTMENT_ID</name>
                            <type>SMALLINT</type>
                            <typeOriginal>NUMBER(4,0)</typeOriginal>
                            <nullable>false</nullable>
                            <description>Primary key column of departments table.</description>
                        </column>
                        <column>
                            <name>DEPARTMENT_NAME</name>
                            <type>VARCHAR(30)</type>
                            <typeOriginal>VARCHAR2(30)</typeOriginal>
                            <nullable>false</nullable>
                            <description>A not null column that shows name of a department. Administration,\u000AMarketing, Purchasing, Human Resources, Shipping, IT, Executive, Public\u000ARelations, Sales, Finance, and Accounting.\u0020</description>
                        </column>
                        <column>
                            <name>MANAGER_ID</name>
                            <type>INT</type>
                            <typeOriginal>NUMBER(6,0)</typeOriginal>
                            <description>Manager_id of a department. Foreign key to employee_id column of employees table. The manager_id column of the employee table references this column.</description>
                        </column>
                        <column>
                            <name>LOCATION_ID</name>
                            <type>SMALLINT</type>
                            <typeOriginal>NUMBER(4,0)</typeOriginal>
                            <description>Location id where a department is located. Foreign key to location_id column of locations table.</description>
                        </column>
                    </columns>
                    <primaryKey>
                        <name>DEPT_ID_PK</name>
                        <column>DEPARTMENT_ID</column>
                    </primaryKey>
                    <foreignKeys>
                        <foreignKey>
                            <name>DEPT_MGR_FK</name>
                            <referencedSchema>HR</referencedSchema>
                            <referencedTable>EMPLOYEES</referencedTable>
                            <reference>
<column>MANAGER_ID</column>
<referenced>EMPLOYEE_ID</referenced>
                            </reference>
                            <deleteAction>RESTRICT</deleteAction>
                            <updateAction>CASCADE</updateAction>
                        </foreignKey>
                        <foreignKey>
                            <name>DEPT_LOC_FK</name>
                            <referencedSchema>HR</referencedSchema>
                            <referencedTable>LOCATIONS</referencedTable>
                            <reference>
<column>LOCATION_ID</column>
<referenced>LOCATION_ID</referenced>
                            </reference>
                            <deleteAction>RESTRICT</deleteAction>
                            <updateAction>CASCADE</updateAction>
                        </foreignKey>
                    </foreignKeys>
                    <rows>27</rows>
                </table>

Complete metadata include definitions for:

• tables
• columns
• descriptions
• primary keys
• foreign keys
• check constraints
• triggers
• views
• procedures/functions (name and parameters but not the code)

Important note: as already mentioned, the tool doesn’t provide fine grained tuning of archiving jobs: extraction scope is based on the security perimeter of the user performing the job, which metadata is unloaded is fixed by the tool and data is fully extracted (no way to filter with utility parameters).

That said, you can use the source database features to configure more fine grained behaviour, for instance by:

• creating a dedicated user
• defining user permissions tailored to the archiving scope defined
• creating views for column or row filtering purposes

To get an idea of how big an archive file can be when exporting a big table, we made a first test with a three columns table containing 26,300,300 rows and whose segment size is 50G: the archive file size is 27G. During the tests, other intermediary sizes were used for the table and each time, the archive file was a little more than 50% of the segment size.

What’s next

In upcoming blog posts, we will dive in the GUI interface and the various challenges encountered on real schemas.

L’article Archiving Oracle databases with SIARD Suite est apparu en premier sur dbi Blog.

What we need:

• Internet access
• Oracle Support account
• new installation of Oracle Linux 8 or 9, RHEL 8 or 9, SLES 15 with enough space for Oracle binaries
• Java JDK 8 to JDK 11
• latest version of AutoUpgrade

Steps to proceed:

• update the system if not already performed
• install OpenJDK if java not yet installed on the system
• install Oracle preinstall package for 26ai
• create oracle base, inventory and home directories with relevant permissions
• change to oracle user created during installation of preinstall package for 26ai
• create AutoUpgrade home with some subdirectories
• download latest AutoUgrade version
• check version of AutoUpgrade
• create AutoUpgrade configuration file for software download
• configure AutoUpgrade keystore to enable access to Oracle Support
• launch software download
• create AutoUpgrade configuration file for Oracle Home creation
• launch Oracle Home creation
• check the newly installed Oracle AI Database 26ai environment

1) Update the system

sudo dnf update

2) Install OpenJDK if java not yet installed on the system

sudo dnf install -y java-11-openjdk

3) Install Oracle preinstall package for 26ai

sudo dnf install -y oracle-database-preinstall-23ai

4) Create oracle base, inventory and home directories with relevant permissions

sudo mkdir -p /u01/app/oracle
sudo mkdir -p /u01/app/oraInventory
sudo mkdir -p /u01/app/oracle/product/23.26.2/dbhome_1
sudo chown -R oracle:oinstall /u01/app/oracle
sudo chown -R oracle:oinstall /u01/app/oraInventory

5) Change to oracle user created during installation of preinstall package for 26ai

sudo su - oracle

6) Create AutoUpgrade home with some subdirectories

mkdir autoupgrade
mkdir autoupgrade/logs
mkdir autoupgrade/patches
mkdir autoupgrade/keystore
mkdir autoupgrade/etc
cd autoupgrade

7) Download latest version of AutoUpgrade

wget -O /home/oracle/autoupgrade/autoupgrade.jar https://download.oracle.com/otn-pub/otn_software/autoupgrade.jar

8) Check version of AutoUpgrade

java -jar autoupgrade.jar -version

9) Create AutoUpgrade configuration file for software download

vi etc/au-download.cfg

global.global_log_dir=/home/oracle/autoupgrade/logs
global.keystore=/home/oracle/autoupgrade/keystore

dl.folder=/home/oracle/autoupgrade/patches
dl.patch=RU,OPATCH
dl.target_version=23
dl.platform=LINUX.X64

10) Configure AutoUpgrade keystore to enable access to Oracle Support

java -jar autoupgrade.jar -config etc/au-download.cfg -patch -load_password
Processing config file ...

Starting AutoUpgrade Patching Password Loader - Type help for available options
Creating new AutoUpgrade Patching keystore - Password required
Enter password:
Enter password again:
AutoUpgrade Patching keystore was successfully created

MOS> add -user <your-mos-account-name>
Enter your secret/Password:
Re-enter your secret/Password:
MOS> save
Convert the AutoUpgrade Patching keystore to auto-login [YES|NO] ? YES
MOS> exit

AutoUpgrade Patching Password Loader finished - Exiting AutoUpgrade Patching

11) Launch software download

java -jar autoupgrade.jar -config etc/au-download.cfg -patch -mode download
AutoUpgrade Patching 26.3.260401 launched with default internal options
Processing config file ...
Loading AutoUpgrade Patching keystore
AutoUpgrade Patching keystore is loaded

Connected to MOS - Searching for specified patches

-----------------------------------------------------
Downloading files to /home/oracle/autoupgrade/patches
-----------------------------------------------------
DATABASE RELEASE UPDATE 23.26.2.0.0
    File: p39099680_230000_Linux-x86-64.zip \ 16%
...

After download has completed, we get the following output:

AutoUpgrade Patching 26.3.260401 launched with default internal options
Processing config file ...
Loading AutoUpgrade Patching keystore
AutoUpgrade Patching keystore is loaded

Connected to MOS - Searching for specified patches

-----------------------------------------------------
Downloading files to /home/oracle/autoupgrade/patches
-----------------------------------------------------
DATABASE RELEASE UPDATE 23.26.2.0.0
    File: p39099680_230000_Linux-x86-64.zip - VALIDATED

OPatch 12.2.0.1.51 for DB 23.0.0.0.0 (Apr 2026)
    File: p6880880_230000_Linux-x86-64.zip - VALIDATED
-----------------------------------------------------

At time of writing, 23.26.2 was just released and Data Pump Bundle Patch could not yet be downloaded for this Release Update.

12) Create AutoUpgrade configuration file for Oracle Home creation

vi etc/au-create-home.cfg

global.global_log_dir=/home/oracle/autoupgrade/logs

crh.folder=/home/oracle/autoupgrade/patches
crh.patch=RU,OPATCH
crh.target_version=23
crh.platform=LINUX.X64
crh.target_home=/u01/app/oracle/product/23.26.2/dbhome_1
crh.home_settings.edition=ee
crh.home_settings.oracle_base=/u01/app/oracle
crh.home_settings.inventory_location=/u01/app/oraInventory
crh.download=no

13) Launch Oracle Home creation

java -jar autoupgrade.jar -config etc/au-create-home.cfg -patch -mode create_home
AutoUpgrade Patching 26.3.260401 launched with default internal options
Processing config file ...
+-----------------------------------------+
| Starting AutoUpgrade Patching execution |
+-----------------------------------------+
Type 'help' to list console commands
patch>lsj -a
patch> +----+-------------+-------+---------+-------+----------+-------+---------------------+
|Job#|      DB_NAME|  STAGE|OPERATION| STATUS|START_TIME|UPDATED|              MESSAGE|
+----+-------------+-------+---------+-------+----------+-------+---------------------+
| 100|create_home_1|EXTRACT|EXECUTING|RUNNING|  09:29:32|37s ago|Extracting gold image|
+----+-------------+-------+---------+-------+----------+-------+---------------------+
Total jobs 1

patch> Job 100 completed
------------------- Final Summary --------------------
Number of databases            [ 1 ]

Jobs finished                  [1]
Jobs failed                    [0]
Jobs restored                  [0]
Jobs pending                   [0]

# Run the root.sh script as root for the following jobs:
For create_home_1 -> /u01/app/oracle/product/23.26.2/dbhome_1/root.sh

# Run the orainstRoot.sh script as root for the following jobs:
For create_home_1 -> /u01/app/oraInventory/orainstRoot.sh

Please check the summary report at:
/home/oracle/autoupgrade/logs/cfgtoollogs/patch/auto/status/status.html
/home/oracle/autoupgrade/logs/cfgtoollogs/patch/auto/status/status.log

To complete the installation, we need to execute with root privileges the two scripts mentioned at the end of the output.

14) Check the newly installed Oracle AI Database 26ai environment

more /home/oracle/autoupgrade/logs/cfgtoollogs/patch/auto/status/status.log
==========================================
   AutoUpgrade Patching Summary Report
==========================================
[Date]           Fri May 01 09:33:13 GMT 2026
[Number of Jobs] 1
==========================================
[Job ID] 100
==========================================
[DB Name]                create_home_1
[Version Before AutoUpgrade Patching] 23.0.0.0.0
[Version After AutoUpgrade Patching]  23.26.2.0.0
------------------------------------------
[Stage Name]    PENDING
[Status]        SUCCESS
[Start Time]    2026-05-01 09:29:32
[Duration]      0:00:00
[Log Directory] /home/oracle/autoupgrade/logs/create_home_1/100/pending
------------------------------------------
[Stage Name]    PREACTIONS
[Status]        SUCCESS
[Start Time]    2026-05-01 09:29:32
[Duration]      0:00:00
[Log Directory] /home/oracle/autoupgrade/logs/create_home_1/100/preaction
------------------------------------------
[Stage Name]    EXTRACT
[Status]        SUCCESS
[Start Time]    2026-05-01 09:29:32
[Duration]      0:01:17
[Log Directory] /home/oracle/autoupgrade/logs/create_home_1/100/extract
------------------------------------------
[Stage Name]    DBTOOLS
[Status]        SUCCESS
[Start Time]    2026-05-01 09:30:51
[Duration]      0:00:00
[Log Directory] /home/oracle/autoupgrade/logs/create_home_1/100/dbtools
------------------------------------------
[Stage Name]    INSTALL
[Status]        SUCCESS
[Start Time]    2026-05-01 09:30:52
[Duration]      0:02:20
[Log Directory] /home/oracle/autoupgrade/logs/create_home_1/100/install
------------------------------------------
[Stage Name]    OH_PATCHING
[Status]        SUCCESS
[Start Time]    2026-05-01 09:33:13
[Duration]      0:00:00
[Log Directory] /home/oracle/autoupgrade/logs/create_home_1/100/opatch
------------------------------------------
[Stage Name]    OPTIONS
[Status]        SUCCESS
[Start Time]    2026-05-01 09:33:13
[Duration]      0:00:00
[Log Directory] /home/oracle/autoupgrade/logs/create_home_1/100/options
------------------------------------------
[Stage Name]    ROOTSH
[Status]        SUCCESS
[Start Time]    2026-05-01 09:33:13
[Duration]      0:00:00
[Log Directory] /home/oracle/autoupgrade/logs/create_home_1/100/rootsh
------------------------------------------
[Stage Name]    DISPATCH
[Status]        SUCCESS
[Start Time]    2026-05-01 09:33:13
[Duration]      0:00:00
[Log Directory] /home/oracle/autoupgrade/logs/create_home_1/100/dispatch
------------------------------------------
[Stage Name]    POSTACTIONS
[Status]        SUCCESS
[Start Time]    2026-05-01 09:33:13
[Duration]      0:00:00
[Log Directory] /home/oracle/autoupgrade/logs/create_home_1/100/postaction
------------------------------------------

sqlplus -V

SQL*Plus: Release 23.26.2.0.0 - Production
Version 23.26.2.0.0

And that’s it.

L’article AutoUpgrade powered easy download and installation of Oracle AI Database 26ai on Linux est apparu en premier sur dbi Blog.

The business requirements to be able to get hands on clones of production databases are nowadays countless:

• developer agility (needing separate and isolated databases),
• production-like shared environment providing same model and code but with a smaller footprint,
• test databases for software update testing,
• data sharing,
• point-in-time reporting, what-if scenario modeling or data audit.

For environments with big databases (tens of terabytes and more), Exascale is very well suited to do the job, with the following main features and advantages:

• source database flexibility: you can now clone read-write and read-only 23/26ai databases, at the PDB and CDB level
• no upstream dependencies: unlike ASM, Exascale no longer requires a test-master read-only copy
• storage reclamation is automatically performed by Exascale for extents no longer referenced by a file
• redirect-on-write techniques enable quick clone creation and maintenance with unprecedented space efficiency
• and last but not least, clone/snapshot is independent of source, which can be dropped without impacting clone

Before going through the details on how to make clones and snapshots, let’s remind what clones and snapshots are.

Snapshot

A snapshot is a read-only point-in-time copy of a file. Possible sources for snapshot creation are regular files, clones or another snapshot.

Clone

A clone is a read-write point-in-time copy of a file, possibly created from a regular file, another clone or a snapshot.

There are actually two types of clones: thin (space efficient) or full (byte-for-byte copy of source).

Cloning implicitly creates snapshots.

How do we create clones or snapshots of PDBs ?

This is where database kernel and Exascale integration really shines: physical storage operations are encapsulated in single SQL commands, thus there is no need to learn new tricky commands and clauses to leverage Exascale cloning and snapshoting capabilities. You use SQL commands you are already familiar with:

CREATE|ALTER PLUGGABLE DATABASE

Creating PDB snapshots

To create a read-only point-in-time reference to a PDB, use:

ALTER PLUGGABLE DATABASE pdb_name SNAPSHOT pdb_snapshot_name;

Remember that snapshots are inconsistent by default (which precisely makes quick creation possible); so cloning a snapshot triggers recovery based on redo data before the new clone can be opened read-write. Thus, required redo data has to be retained or restored from backup.

The DBA_PDB_SNAPSHOTS dictionary view is available to display snapshots information.

Creating PDB thin clones

A PDB thin clone is a space-efficient clone of an existing PDB. You need to use the SNAPSHOT COPY clause like in the following syntax:

CREATE PLUGGABLE DATABASE pdb_name FROM source_pdb_name SNAPSHOT COPY;

Thin cloning a PDB

CREATE PLUGGABLE DATABASE thin_clone_pdb_name FROM source_pdb_name SNAPSHOT COPY;
ALTER PLUGGABLE DATABASE thin_clone_pdb_name OPEN INSTANCES=ALL;

Thin cloning a PDB snapshot

CREATE PLUGGABLE DATABASE thin_clone_pdb_name FROM source_pdb_name USING SNAPSHOT pdb_snapshot_name SNAPSHOT COPY;
ALTER PLUGGABLE DATABASE thin_clone_pdb_name OPEN INSTANCES=ALL;

Creating PDB full clones

As already mentioned, a full clone is a byte-for-byte copy of an existing PDB and therefore takes the same storage space than source PDB. The command to be used is a simple:

CREATE PLUGGABLE DATABASE … FROM …

The missing SNAPSHOT COPY clause directs the database to create a complete block-for-block copy of the PDB.

CREATE PLUGGABLE DATABASE full_clone_pdb_name FROM source_pdb_name;
ALTER PLUGGABLE DATABASE full_clone_pdb_name OPEN INSTANCES=ALL;

Existing snapshots or thin clones can also be referenced as a source in the preceding commands.

What about cloning an entire container database ?

Space-efficient clones of an entire container database are possible thanks to the powerful gDBClone utility written by Ruggero Citton, to whom we owe other great utilities ODABR (consistent and incremental backup and restore for Oracle Database Appliance nodes), KVMBR (backup and restore of KVM domains) or more recently DSCM & DSCMREST (deployment of Oracle database containers with efficient storage snapshots).

The gDBClone utility enables:

• End-to-end thin and full cloning of a CDB with all its PDB
• File clone operations
• RMAN calls to make cloned files consistent
• Instance related files (such as password, spfile or TDE wallet files) creation
• GI registration

Here are some simple examples to respectively thin clone and full clone an entire CDB:

gDBClone.bin snap –sdbname source_db_name –tdbname target_db_name

gDBClone.bin clone –sdbname source_db_name –tdbname target_db_name

More on this powerful tool in Exascale documentation and Oracle Support KB145187 article.

Performance

Thin cloning a 2.4 TB PDB takes less than 12 seconds while opening the new thin clone on a two instances cluster database takes another 11 seconds, for a total elapsed time of the entire operation under 30 seconds !!!

Additional details can be found in this Alex Blyth blog post.

Conclusion

By leveraging state-of-the-art redirect-on-write technologies, Exascale is a leap forward and game changing technology when it comes to clone and snapshot features and capabilities. It is CI/CD friendly and the ideal tool to address the business requirements for test and qualification environments by letting DBAs provide very quickly space efficient copies of production or standby databases or PDBs.

Give it a try to be definitely convinced.

L’article Exascale new clone and snapshot capabilities est apparu en premier sur dbi Blog.

In this blog post, we will explore the Exascale storage architecture and processes pertaining to Exascale. But before diving in, a quick note to avoid any confusion about Exascale: even though in the previous article, we focused on Exascale Infrastructure and its various benefits in terms of small footprint and hyper-elasticity based on modern cloud characteristics, it is important to keep in mind that Exascale is an Exadata technology and not a cloud-only technology. You can benefit from it in non-cloud deployments as well such as on Exadata Database Machine deployed in your data centers.

Exascale storage components

Here is the overall picture:

Exascale cluster

An Exascale Cluster is composed of Exadata storage servers to provide storage to Grid Infrastructure clusters and the databases. An Exadata storage server can belong to only one Exascale cluster.

Software services (included in the Exadata System Software stack) run on each Exadata cluster for managing the cluster resources made available to GI and databases, namely pool disks, storage pools, vaults, block volumes and many other.

With Exadata Database Service on Exascale Infrastructure, the number of Exadata storage servers included in the Exascale cluster can be quite huge, hundreds maybe even thousands storage servers, to enable cloud-scale storage resource pooling.

Storage pools

A storage pool is a collection of pool disks (see below for details about pool disks).

Each Exascale cluster requires at least one storage pool.

A storage pool can be dynamically reconfigured by changing pool disks size, allocating more pool disks or adding Exadata storage servers.

The pool disks found inside a storage pool must be of the same media type.

Pool disks

A pool disk is physical storage space allocated from an Exascale-specific cell disk to be integrated in a storage pool.

Each storage server physical disk has a LUN in the storage server OS and a cell disk is created as a container for all Exadata-related partitions within the LUN. Each partition in a cell disk is then designated as a pool disk for Exascale (or grid disk in case of ASM is used instead of Exascale).

A media type is associated to each pool disk based on the underlying storage device and can be one of the following :

• HC : points to high capacity storage using hard disk drives
• EF : based on extreme flash storage devices
• XT : corresponds to extended storage using hard disk drives found in Exadata Storage Server X11M Extended (XT) hardware

Vaults

Vaults are logical storage containers used for storing files and are allocated from storage pools. By default, without specific provisioning attributes, vaults can use all resources from all storage pools of an Exascale cluster.

Here are the two main services provided by vaults:

1. Security isolation: a security perimeter is associated with vaults based on user access controls which guarantee a strict isolation of data between users and clusters.
2. Resource control: storage pool resources usage is configured at the vault level for attributes like storage space, IOPS, XRMEM and flash cache sizing.

For those familiar with ASM, think of vaults as the equivalent to ASM disk groups. For example, instance parameters like ‘db_create_file_dest’ and ‘db_recovery_file_dest’ reference vaults instead of ASM disk groups by using ‘@vault_name’ syntax.

Since attributes like redundancy, type of file content, type of media are positioned at the file level instead at the logical storage container, there is no need to organize vaults in the same manner as we did for disk groups. For instance, we don’t need to create a first vault for data and a second vault for recovery files as we are used to with ASM.

Beside database files, vaults can also store other types of files even though it is recommended to store non database files on block volumes. That’s because Exascale is optimized for storing large files such as database files whereas regular files are typically much smaller and fit more on block volumes.

Files

The main files found on Exascale storage are Database and Grid Infrastructure files. Beyond that, all objects in Exascale are represented as files of a certain type. Each file type has storage attributes defined in a template. The file storage attributes are :

• mediaType : HC, EF or XT
• redundancy : currently high
• contentType : DATA and RECO

This makes a huge difference with ASM where different redundancy needs required to create different disk groups. With Exascale, it is now possible to store files with different redundancy requirements in the same storage structure (vaults). This also enables to optimize usage of the storage capacity.

Files are composed of extents of 8MB in size which are mirrored and stripped across all vault’s storage resources.

The tight integration of the database kernel with Exascale makes it possible for Exascale to automatically understand the type of file the database asks to create and thus applies the appropriate attributes defined in the file template. This prevents Exascale to store data and recovery files extents (more on extents in the next section) on the same disks and also guarantees that mirrored extents are located on different storage servers than the primary extent.

File extents

Remember that in Exascale, storage management moved from the compute servers to the storage servers. Specifically, this means the building blocks of files, namely extents, are managed by the storage servers.

The new data structure used for extent management is a mapping table which tracks for each file extent the location of the primary and mirror copy extents in the storage servers. This mapping table is cached by each database server and instance to retrieve its file extents location. Once the database has the extent location, it can directly make an I/O call to the appropriate storage server. In case the mapping table is no more up-to-date because of database physical structure changes or storage servers addition or removal, an I/O call can be rejected, triggering a mapping table refresh allowing the I/O call to be retried.

Exascale Block Store with RDMA-enabled block volumes

Block volumes can be allocated from storage pools to store regular files on file systems like ACFS or XFS. They also enable centralization of VC VM images, thus cutting the dependency of VM images to internal compute node storage and streamlining migrations between physical database servers. Clone, snapshot and backup and restore features for block volumes can leverage all resources of the available storage servers.

Exascale storage services

From a software perspective, Exascale is composed of a number of software services available in the Exadata System Software (since release 24.1). These software services run mainly on the Exadata Storage Servers but also on the Exadata Database Servers.

Exascale storage server services

Exascale database server services

Below diagram depicts how the various Exascale services are dispatched on the storage and compute nodes:

Wrap-up

By rearchitecting Exadata storage management with focus on space efficiency, flexibility and elasticity, Exascale can now overcome the main limitations of ASM:

• diskgroup resizing complexity and time-consuming rebalance operation
• space distribution among DATA and RECO diskgroups requiring rigorous estimation of storage needs for each
• sparse diskgroup requirement for cloning with a read-only test master or use of ACFS without smart scan
• redundancy configuration at the diskgroup level

The below links will provide you further details on the matter:

Oracle Exadata Exascale blog

Oracle Exadata Exascale advantages blog

Oracle Exadata Exascale storage fundamentals blog

Oracle Exascale documentation

Oracle Exadata architecture

Oracle And Me blog – Why ASM Needed an Heir Worthy of the 21st Century

Oracle And Me blog – New Exascale architecture

More on the exciting Exascale technology in coming posts …

L’article Exascale storage architecture est apparu en premier sur dbi Blog.

Introduction

Exadata on Exascale Infrastructure is a new deployment option for Exadata Database Service. It comes in addition to the well known Exadata Cloud@Customer or Exadata on Dedicated Infrastructure options already available. It is based on new storage management technology decoupling database and Grid Infrastructure clusters from the Exadata underlying storage servers by integrating the database kernel directly with the Exascale storage structures.

What is Exascale Infrastructure ?

Simply put, Exascale is the next-generation of Oracle Exadata Database Services. It combines a cloud storage approach for flexibilty and hyper-elasticity with the performance of Exadata Infrastructure. It introduces a loosely-coupled shared and multitenant architecture where Database and Grid Infrastructure clusters are decoupled from the underlying Exadata storage servers which become a pool of shared storage resources available for multiple Grid Infrastructure clusters and databases.

Strict data isolation provides secure storage sharing while storage pooling enables flexible and dynamic provisioning combined with better storage space and processing utilization.

Advanced snapshot and cloning features, leveraging Redirect-On-Write technology instead of Copy-On-Write, enable space-efficient thin clones from any read/write database or pluggable database. Read-only test master are therefore a thing of the past. These features alone make Exascale a game-changer for database refreshes, CI/CD workflows and pipelines, development environments provisioning, all done with single SQL commands and much faster than before.

Block storage services allow the creation of arbitrary-sized block volumes for use by numerous applications. Exascale block volumes are also used to store Exadata database server virtual machines images enabling :

• creation of more virtual machines
• removal of local storage dependency inside the Exadata compute nodes
• seamless migration of virtual machines between different compute nodes

Finally, the following hardware and software considerations complete this brief presentation of Exascale:

• runs on 2-socket Oracle Exadata system hardware with RoCE Network Fabric (X8M-2 or later)
• Database 23ai release 23.5.0 or later is required for full-featured native Database file storage in Exascale
• Exascale block volumes support databases using older Database software releases back to Oracle Database 19c
• Exascale is built into Exadata System Software (since release 24.1)

Architecture

The main point with Exascale architecture is cloud-scale, multi-tenant resource pooling, both for storage and compute.

Storage pooling

Exascale is based on pooling storage servers which provide services such as Storage Pools, Vaults and Volume Management. Vaults, which can be considered an equivalent of ASM diskgroups, are directly accessed by the database kernel.

File and extent management is done by Exadata System Software, thus freeing the compute layer from ASM processes and memory structures for database files access and extents management (with Database 23ai or 26ai; for Database 19c, ASM is still required). With storage management moving to storage servers, resource management becomes more flexible and more memory and CPU resources are available on compute nodes to process database tasks.

Exascale also provides redundancy, caching, file metadata management, snapshots and clones as well as security and data integrity features.

Of course, since Exascale is built on top of Exadata, you benefit from features like RDMA, RoCE, Smart Flash Cache, XRMEM, Smart Scan, Storage Indexes, Columnar Caching.

Compute pooling

On the compute side, we have database-optimized servers which run Database 23ai or newer and Grid Infrastructure Cluster management software. The physical database servers host the VM Clusters and are managed by Oracle. Unlike Exadata on Dedicated Infrastructure, there is no need to provision an infrastructure before going ahead with VM Cluster creation and configuration : in Exascale, you only deal with VM Clusters.

VM file systems are centrally-hosted by Oracle on RDMA-enabled block volumes in a system-vault. VM images used by the VM Clusters are no more hosted on local storage on the database servers. This enables the number of VMs running on the database servers to raise from 12 to 50.

Each VM Cluster accesses a Database Vault storing the database files with strict isolation from other VM Clusters database files.

Virtual Cloud Network

Client and backup connectivity is provided by Virtual Cloud Network (VCN) services.

This loosely-coupled, shared and multitenant architecture enables far greater flexibilty than what was possible with ASM or even Exadata on Dedicated Infrastructure. Exascale’s hyper-elasticity enables to start with very small VM Clusters and then scale as the workloads increase, with Oracle managing the infrastructure automatically. You can start as small as 1 VM per cluster (up to 10), 8 eCPUs per VM (up to 200), 22GB of memory per VM, 220GB file system storage per VM and 300GB Vault storage per VM Cluster (up to 100TB). Memory is tightly coupled to eCPUs configuration with 2.75GB per eCPU and thus does not scale independently from the eCPUs number.

For those new to eCPU, it is a standard billing metric based on the number of cores per hour elastically allocated from a pool of compute and storage servers. eCPUs are not tied to the make, model or clock speed of the underlying processor. By contrast, an OCPU is the equivalent of one physical core with hyper-threading enabled.

To summarize

To best understand what Exascale Infrastructure is and introduces, here is the wording which best describes this new flavor of Exadata Database Service :

• loosely-coupling of compute and storage cloud
• hyper-elasticity
• shared and multi-tenant service model
• ASM-less (for 23ai or newer)
• Exadata performance, reliability, availability and security at any scale
• CI/CD friendly
• pay-per-use model

Stay tuned for more on Exascale …

L’article Exascale Infrastructure : new flavor of Exadata Database Service est apparu en premier sur dbi Blog.