Oracle Data Pump can import simple programs without arguments, but it fails when handling Scheduler-Programs that contain arguments. This has been a persistent issue with no official resolution.
Since this was a critical challenge in a recent project of mine, I’d like to share my solution.

In rare cases, the scripts are available and can be reused, but when you need to dig through Change Requests, JIRA tickets or the Document Management System (DMS) because the developer has already retired, you’re often left with no choice but to reengineer the solution.

At first, I started with a schema-export and import which generated a log file with 15,000 lines and hundreds of error messages. I was puzzled by this message:
There were more of them, however I would like to use this single Scheduler-Program as example.

Here’s an example of an error message I encountered:

19-OCT-24 10:55:09.222: ORA-39083: Object type PROCOBJ:"MMTK"."MAIL_MMTK_PRG" failed to create with error:
ORA-06550: line 8, column 265:
PL/SQL: ORA-00942: table or view does not exist
ORA-06550: line 8, column 21:
PL/SQL: SQL Statement ignored
ORA-06550: line 11, column 5:
PLS-00364: loop index variable 'ARG' use is invalid
ORA-06550: line 11, column 2:
PL/SQL: Statement ignored

Failing sql is:
BEGIN 
dbms_scheduler.create_program('"MAIL_MMTK_PRG"','STORED_PROCEDURE',
'PKG_JOBMMTK.prc_wmail'
,18, FALSE,
'Send Mail'
);
DECLARE 
CURSOR prog_args IS SELECT replace(a.argument_name, '''', '''''') arg_name,  a.argument_position position, a.argument_type type_name, a.metadata_attribute int_arg_type, a.default_anydata_value value, decode(a.out_argument,'FALSE',0,'TRUE',1) out_flag  FROM SYSTEM.SCHEDULER_PROGRAM_ARGS_TMP a  where a.owner='MMTK' and  a.program_name='MAIL_MMTK_PRG';
 BEGIN
FOR arg in prog_args LOOP
 IF arg.int_arg_type IS NULL THEN
 dbms_scheduler.define_anydata_argument('"MAIL_MMTK_PRG"' , arg.position,  CASE WHEN arg.arg_name IS NULL THEN NULL ELSE 
     '"'||arg.arg_name||'"' END, arg.type_name, arg.value, arg.out_flag=1 );
 ELSE
 dbms_scheduler.define_metadata_argument('"MAIL_MMTK_PRG"' ,arg.int_arg_type, arg.position, CASE WHEN arg.arg_name IS NULL THEN NULL ELSE 
     '"'||arg.arg_name||'"' END );
 END IF;
 END LOOP;
 END;
dbms_scheduler.enable('"MAIL_MMTK_PRG"');COMMIT; END;

The first hint (ORA-00942) suggests a missing table or privilege, but which one?
Maybe the SYSTEM.SCHEDULER_PROGRAM_ARGS_TMP in the DECLARE-block? It does not exist. Sounds like a temporary view during import.
The second hint about variable ‘ARG’ is mysterious – though the loop variable appears syntactically correct, it still triggers an error.
Furthermore, in another Scheduler-Program, I found the same error.
Although the Scheduler-Programs worked fine in the existing environment, I could not import them. Missing tables or permissions as the reason for the errors could be excluded after some work.
Somewhere on the internet, I found the information, that this is an unsolved problem. A request for enhancement is pending …

Now, how can we script these Scheduler-Programs?
dbms-metadata? Good idea! Let’s see what it delivers:

To extract the definition of the Scheduler-Program, I used the following command:

select dbms_metadata.get_ddl('PROCOBJ','MAIL_MMTK_PRG','MMTK') from dual;

BEGIN
  dbms_scheduler.create_program('MAIL_MMTK_PRG','STORED_PROCEDURE',
  'PKG_JOBMMTK.prc_wmail', 18, FALSE, 'Send Mail'
);
COMMIT;
END;
/

Nice
It retrieves the Scheduler-Program itself, but not its arguments
This is just the “failing sql” we have seen in the error message above.

Using USER_SCHEDULER_PROGRAM_ARGS allows us to reconstruct missing arguments

select argument_position, argument_name, argument_type, default_value 
 from USER_SCHEDULER_PROGRAM_ARGS 
 where  program_name = 'MAIL_MMTK_PRG' order by   argument_position;

ARGUMENT_POSITION ARGUMENT_NAME      ARGUMENT_TYPE  DEFAULT_VALUE
----------------- ------------------ -------------- --------------------
                1 P_HOST             VARCHAR2       mailhost
                2 P_DOMAIN           VARCHAR2       mycompany.com
                3 P_PORTNUM          NUMBER         25
                4 P_FROM             VARCHAR2
                5 P_TO               VARCHAR2
                6 P_CC               VARCHAR2
                7 P_BCC              VARCHAR2
                8 P_REPLYTO          VARCHAR2
                9 P_SUBJECT          VARCHAR2
               10 P_MSGTXT           VARCHAR2
               11 P_CONTENTS         VARCHAR2
               12 P_ATTACH           VARCHAR2
               13 P_URL              VARCHAR2
               14 P_TRACEMODE        VARCHAR2       N
               15 P_MIMETYPE         VARCHAR2       text/plain
               16 P_DISPOSITION      VARCHAR2       attachment
               17 P_CHARSET          VARCHAR2       iso-8859-1
               18 P_TRANSFERTENCODE  VARCHAR2

I was not aware that Emails need to have 18 arguments, but ok
Now I started puzzling pieces. The syntax for an argument is:

dbms_scheduler.define_program_argument(
 program_name      => 'MAIL_MMTK_PRG',
 argument_position => 1,
 argument_name     => 'P_HOST',
 argument_type     => 'VARCHAR2',
 default_value     => 'mailhost'  );

copy this block 18 times and insert the values from listing above.
Yes, I could have done some clever dynamic-SQL …

At the end, the entire code block looks like that:
Before, you need to create the scheduler-program as seen above – of course …

begin
  dbms_scheduler.define_program_argument(
    program_name      => 'MAIL_MMTK_PRG',
    argument_position => 1,
    argument_name     => 'P_HOST',
    argument_type     => 'VARCHAR2',
    default_value     => 'mailhost'  );
  dbms_scheduler.define_program_argument(
    program_name      => 'MAIL_MMTK_PRG',
    argument_position => 2,
    argument_name     => 'P_DOMAIN',
    argument_type     => 'VARCHAR2',
    default_value     => 'mycompany.com'  );
  dbms_scheduler.define_program_argument(
    program_name      => 'MAIL_MMTK_PRG',
    argument_position => 3,
    argument_name     => 'P_PORTNUM',
    argument_type     => 'NUMBER',
    default_value     => 25  );
  dbms_scheduler.define_program_argument(
    program_name      => 'MAIL_MMTK_PRG',
    argument_position => 4,
    argument_name     => 'P_FROM',
    argument_type     => 'VARCHAR2',
    default_value     => NULL  );
  dbms_scheduler.define_program_argument(
    program_name      => 'MAIL_MMTK_PRG',
    argument_position => 5,
    argument_name     => 'P_TO',
    argument_type     => 'VARCHAR2',
    default_value     => NULL  );
  dbms_scheduler.define_program_argument(
    program_name      => 'MAIL_MMTK_PRG',
    argument_position => 6,
    argument_name     => 'P_CC',
    argument_type     => 'VARCHAR2',
    default_value     => NULL  );
  dbms_scheduler.define_program_argument(
    program_name      => 'MAIL_MMTK_PRG',
    argument_position => 7,
    argument_name     => 'P_BCC',
    argument_type     => 'VARCHAR2',
    default_value     => NULL  );
  dbms_scheduler.define_program_argument(
    program_name      => 'MAIL_MMTK_PRG',
    argument_position => 8,
    argument_name     => 'P_REPLYTO',
    argument_type     => 'VARCHAR2',
    default_value     => NULL  );
  dbms_scheduler.define_program_argument(
    program_name      => 'MAIL_MMTK_PRG',
    argument_position => 9,
    argument_name     => 'P_SUBJECT',
    argument_type     => 'VARCHAR2',
    default_value     => NULL  );
  dbms_scheduler.define_program_argument(
    program_name      => 'MAIL_MMTK_PRG',
    argument_position => 10,
    argument_name     => 'P_MSGTXT',
    argument_type     => 'VARCHAR2',
    default_value     => NULL  );
  dbms_scheduler.define_program_argument(
    program_name      => 'MAIL_MMTK_PRG',
    argument_position => 11,
    argument_name     => 'P_CONTENTS',
    argument_type     => 'VARCHAR2',
    default_value     => NULL  );
  dbms_scheduler.define_program_argument(
    program_name      => 'MAIL_MMTK_PRG',
    argument_position => 12,
    argument_name     => 'P_ATTACH',
    argument_type     => 'VARCHAR2',
    default_value     => NULL  );
  dbms_scheduler.define_program_argument(
    program_name      => 'MAIL_MMTK_PRG',
    argument_position => 13,
    argument_name     => 'P_URL',
    argument_type     => 'VARCHAR2',
    default_value     => NULL  );
  dbms_scheduler.define_program_argument(
    program_name      => 'MAIL_MMTK_PRG',
    argument_position => 14,
    argument_name     => 'P_TRACEMODE',
    argument_type     => 'VARCHAR2',
    default_value     => 'N'  );
  dbms_scheduler.define_program_argument(
    program_name      => 'MAIL_MMTK_PRG',
    argument_position => 15,
    argument_name     => 'P_MIMETYPE',
    argument_type     => 'VARCHAR2',
    default_value     => 'text/plain'  );
  dbms_scheduler.define_program_argument(
    program_name      => 'MAIL_MMTK_PRG',
    argument_position => 16,
    argument_name     => 'P_DISPOSITION',
    argument_type     => 'VARCHAR2',
    default_value     => 'attachment'  );
  dbms_scheduler.define_program_argument(
    program_name      => 'MAIL_MMTK_PRG',
    argument_position => 17,
    argument_name     => 'P_CHARSET',
    argument_type     => 'VARCHAR2',
    default_value     => 'iso-8859-1'  );
  dbms_scheduler.define_program_argument(
    program_name      => 'MAIL_MMTK_PRG',
    argument_position => 18,
    argument_name     => 'P_TRANSFERTENCODE',
    argument_type     => 'VARCHAR2',
    default_value     => NULL  );
commit;
end;
/

exec DBMS_SCHEDULER.ENABLE(name=>'MAIL_MMTK_PRG');

I hope this guide helps others facing the same issue. If you’ve encountered this problem before or have alternative solutions, share your thoughts in the comments!

L’article Why Data Pump Fails to Import Scheduler-Programs est apparu en premier sur dbi Blog.

Do you suffer from the fact that release updates take longer each time?
Are you afraid that your hard disk will fill up because the Oracle-Home is getting bigger and bigger?
Here is the how-to for cleaning up your environment.

A) cleanup – option 1 for Out-Of-Place patching

According to Oracle’s recommendations, you did “out of place patching” and have now several Oracle-Home directories.
All your databases are using the new ORACLE_HOME (OH) and the old OHs are now obsolete.
Cleaning up is not only a matter of disk space. The old Software releases contain security issues – of course. This is the very reason for patching.

Set the environment for the OH you want to delete

export ORACLE_HOME=/u01/app/oracle/product/19.25.0/dbhome_1
PATH=$ORACLE_HOME/bin:$ORACLE_HOME/OPatch:$ORACLE_HOME/perl/bin:$ORACLE_HOME/ctx/bin
PATH=$PATH:/home/oracle/.local/bin:/home/oracle/bin:/usr/local/bin:/usr/bin:/usr/ccs/bin:/usr/local/sbin:/usr/sbin:/sbin
export PATH
export CV_ASSUME_DISTID=OEL8.1

cd $ORACLE_HOME/deinstall
./deinstall
   -> enter required answers …
       Enter     (confirm [LISTENER] )
       Enter     (confirm databases [] )
        Y        (to confirm to delete the listed OH)

you can delete the empty directory, now

cd /u01/app/oracle/product/
du -sh *
rmdir 19.25.0

B) cleanup – option 2 for Out-Of-Place patching

In some cases, the deinstall does not work and gives you some error messages.
Now we have the option to detach AND remove the old ORACLE_HOME.
In this example our OH-path is: /u01/app/oracle/product/19.15.0.0.220419.EE

first you need to find the “Home-Name” for the OH to deinstall – you might have to look into several subfolders for it

cd /u01/app/oraInventory/logs
ls -l
cd InstallActions<timestamp>
grep -i ORACLE_HOME_NAME *.log
-->> responds for example with:   OraDB19Home1

set the environment for the ORACLE_HOME to remove and DETACH it

export ORACLE_HOME=/u01/app/oracle/product/19.15.0.0.220419.EE
PATH=$ORACLE_HOME/bin:$ORACLE_HOME/OPatch:$ORACLE_HOME/perl/bin:$ORACLE_HOME/ctx/bin
PATH=$PATH:/home/oracle/.local/bin:/home/oracle/bin:/usr/local/bin:/usr/bin:/usr/ccs/bin:/usr/local/sbin:/usr/sbin:/sbin
export PATH
export CV_ASSUME_DISTID=OEL8.1

cd $ORACLE_HOME/oui/bin
./runInstaller -silent -detachHome ORACLE_HOME="/u01/app/oracle/product/19.15.0.0.220419.EE" ORACLE_HOME_NAME="OraDB19Home1"

cd /u01/app/oracle/product
rm -rf 19.15.0.0.220419.EE

and don’t forget to remove the obsolete OH from the oratab file

vi /etc/oratab

C) cleanup – option for IN-Place patching

For reasons, you could not do “out of place” patching. When you have an OH which was already patched several times, the OH increased in size over time. The old patches got stored in a subdirectory for rollback reasons.

Simply check the size of the subdirectory

du -sh $ORACLE_HOME/.patch_storage/
-->> 6.6G  .patch_storage/

I checked this for an ORACLE_HOME with up to 7 RUs installed:

.patch_storage with   2 RUs     3 RUs     4 RUs     5 RUs     6 RUs     7 RUs
          Size GB      5.6       6.6        9.1       12        14        18

you can imagine that each run of opatch needs more and more time for a larger repository in directory ‘.patch_storage’.
For the first patch it goes ‘on the fly’. The second needs a coffee break, the third needs a lunch break …

with this command you can list your obsolete patches

opatch util listorderedinactivepatches

now delete them (this will take a while) and it leaves intentionally the last RU in the repository

opatch util deleteinactivepatches

the disk usage is now reduced, and the next patch will run definitively faster

du -sh $ORACLE_HOME/.patch_storage/
-->> 6.0G .patch_storage/

I am an impatient DBA and I don’t like drinking lots of coffee while waiting for opatch to finish. This is why I love to keep my Oracle-Home directories small and smart.

D) add on – patch the OPatch

regardless of you do in-place or out-of-place patching, you need to update your opatch with the most recent release. The very most DBAs follow the advice in p6880880_opatch_README_12.2.0.1.xy.txt
“Please take a backup of ORACLE_HOME/OPatch into a dedicated backup location” and rename the folder OPatch to OPatch_old.
With the next RU, they move the current to OPatch_even_older or some other funny name.
Just to remember: the purpose of patching is to REPLACE older Software, which contains security issues, by newer releases, containing fixes.
Did you know that opatch contains its own Java? So why should we leave an outaged Java on the system as a gate for hackers?

Remove it immediately after you have verified, the new release is working.

--->>> rm -rf ORACLE_HOME/OPatch_old

I hope, this helps you to keep your environment clean and tidy.

L’article Cleanup Oracle After Patching est apparu en premier sur dbi Blog.

In today’s data-driven world, organizations often leverage a variety of databases to meet their diverse data storage and processing needs. MongoDB and Oracle Database are two prominent names in the database landscape, each excelling in different areas. MongoDB is renowned for its flexibility and scalability in handling unstructured data, while Oracle Database is a powerhouse for structured data with robust transactional capabilities.

Integrating MongoDB documents within an Oracle database allows organizations to harness the strengths of both systems. This hybrid approach enables seamless data management, providing the flexibility of NoSQL with the reliability and performance of a traditional RDBMS. In this blog, we will explore eth technical steps involved in integrating MongoDB documents into an Oracle database.

Why Integrate MongoDB Documents in Oracle Database ?

In short, is for getting the best of two worlds.

1. Unified Data View: Combining MongoDB’s unstructured data with Oracle’s structured data creates a unified view, making it easier to perform comprehensive data analysis and reporting.

2. Enhanced Flexibility: MongoDB’s schema-less architecture allows for rapid development and iteration, while Oracle’s structured schema ensures data integrity and consistency.

3. Scalability and Performance: MongoDB’s horizontal scalability complements Oracle’s vertical scaling capabilities, providing a balanced approach to handling large volumes of data.

4. Cost Efficiency: By leveraging both databases, organizations can optimize costs by storing data where it is most efficient based on its nature and usage patterns.

Technical steps to connect the two worlds.

The VM preparation

I made some changes to the configuration to make the database changes persistent.

For this example I use an Oracle 23 AI docker container. How to use it is explained here: https://www.oracle.com/my/database/free/get-started/

The VM used is an Oracle Linux Server VM deployed on OCI:

[opc@db23ai ~]$ cat /etc/os-release
NAME="Oracle Linux Server"
VERSION="9.4"

The docker part installation is described here: https://docs.docker.com/engine/install/rhel/

[root@db23ai ~]# yum install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

Next,I create the oracle user and groups with the same ID’s as used in the container:

[root@db23ai ~]# groupadd -g 54321 oinstall
[root@db23ai ~]# groupadd -g 54322 dba
[root@db23ai ~]# groupadd -g 54323 oper
[root@db23ai ~]# useradd -g oinstall -G dba,oper,docker --uid 54321 -m oracle

Finally I install the sqlplus client and mongo shell:

[root@db23ai u02]# dnf install oracle-instantclient-sqlplus.x86_64
[root@db23ai u02]# wget https://downloads.mongodb.com/compass/mongodb-mongosh-2.2.12.x86_64.rpm
[root@db23ai u02]# rpm -ivh mongodb-mongosh-2.2.12.x86_64.rpm
warning: mongodb-mongosh-2.2.12.x86_64.rpm: Header V4 RSA/SHA256 Signature, key ID d361cb16: NOKEY
Verifying...                          ################################# [100%]
Preparing...                          ################################# [100%]
Updating / installing...
   1:mongodb-mongosh-2.2.12-1.el8     ################################# [100%]
...

The Oracle 23 AI container

The port 1700 is for database sqlplus connections and the port 27017 is for mongosh connections. I exported some volumes first to make the Oracle database persistent, and to have a place to install ORDS (https://www.oracle.com/ch-de/database/technologies/appdev/rest.html)

Get the container:

[root@db23ai ~]# sudo su - oracle
[oracle@db23ai ~]$ docker pull container-registry.oracle.com/database/free:latest

Run the container:

[oracle@db23ai ~]$ mkdir /u02/data/DB
[oracle@db23ai ~]$ mkdir /u02/reco/DB
[oracle@db23ai ~]$ mkdir /u02/ords

[oracle@db23ai ~]$ docker run  -it --name 23ai -p 1700:1521 -p 27017:27017 \
-v /u02/data/DB:/opt/oracle/oradata \
-v /u02/reco/DB:/opt/oracle/reco \
-v /u02/oracle/ords:/opt/oracle/ords \
-e ENABLE_ARCHIVELOG=true  \
-e ORACLE_PWD="Hello-World-123" \
container-registry.oracle.com/database/free
...

# Once the traces shows that database is running we can start it as daemon (-d option)

# Test the connection to the PDB 
[oracle@db23ai ~]$ sqlplus system/"Hello-World-123"@localhost:1700/freepdb1

SQL*Plus: Release 23.0.0.0.0 - Production on Thu Jul 11 12:09:40 2024
Version 23.4.0.24.05

Copyright (c) 1982, 2024, Oracle.  All rights reserved.

Last Successful login time: Thu Jul 11 2024 11:59:21 +00:00

Connected to:
Oracle Database 23ai Free Release 23.0.0.0.0 - Develop, Learn, and Run for Free
Version 23.4.0.24.05
SQL> show con_id

CON_ID
------------------------------
3

Create the scott schema:

SQL> drop user scott cascade;

SQL> create user SCOTT identified by tiger default tablespace users;

SQL> grant connect, create session, create table, create view, create sequence, create procedure, create job, to scott; 

SQL> alter user scott quota unlimited on users;

Install ORDS to give access to mongo shell to the database

Download ORDS from https://www.oracle.com/database/sqldeveloper/technologies/db-actions/download/

Download it to /u02/oracle/ords/ords_pkg wich is the /u02/oracle/ords directory in the docker container.

Connect as oracle user to the container and install ORDS:

# connect to the container
[oracle@db23ai ~]$ docker exec -u 54321 -it 23ai /bin/bash

bash-4.4$ id
uid=54321(oracle) gid=54321(oinstall) groups=54321(oinstall),54322(dba),54323(oper),54324(backupdba),54325(dgdba),54326(kmdba),54330(racdba)

bash-4.4$ cd /opt/oracle/ords/ords_pkg
bash-4.4$ ls
ords-24.2.2.187.1943.zip

bash-4.4$ unzip ords-24.2.2.187.1943.zip
...

bash-4.4$ export JAVA_HOME=/opt/oracle/product/23ai/dbhomeFree/jdk
bash-4.4$ export PATH=$PATH:/opt/oracle/product/23ai/dbhomeFree/jdk/bin

bash-4.4$ /opt/oracle/ords/ords_pkg/bin/ords --config /opt/oracle/ords install

ORDS: Release 24.2 Production on Thu Jul 11 12:20:40 2024

Copyright (c) 2010, 2024, Oracle.

Configuration:
  /opt/oracle/ords

The configuration folder /opt/oracle/ords does not contain any configuration files.

Oracle REST Data Services - Interactive Install

  Enter a number to select the TNS net service name to use from /opt/oracle/product/23ai/dbhomeFree/network/admin/tnsnames.ora or specify the database connection
    [1] EXTPROC_CONNECTION_DATA SID=PLSExtProc
    [2] FREE         SERVICE_NAME=FREE
    [3] FREEPDB1     SERVICE_NAME=FREEPDB1
    [S] Specify the database connection
  Choose [1]: 3 <<<<<<<<<<<<< MY ENTRY <<<<<<<<<<<<<<<<
  Provide database user name with administrator privileges.
    Enter the administrator username: sys. <<<<<<<<<<<<< MY ENTRY <<<<<<<<<<<<<<<<
  Enter the database password for SYS AS SYSDBA:  Hello-World-123 <<<<<<<<<<<<< MY ENTRY <<<<<<<<<<<<<<<<

Retrieving information.
ORDS is not installed in the database. ORDS installation is required.

  Enter a number to update the value or select option A to Accept and Continue
    [1] Connection Type: TNS
    [2] TNS Connection: TNS_NAME=FREEPDB1 TNS_FOLDER=/opt/oracle/product/23ai/dbhomeFree/network/admin
           Administrator User: SYS AS SYSDBA
    [3] Database password for ORDS runtime user (ORDS_PUBLIC_USER): <generate>
    [4] ORDS runtime user and schema tablespaces:  Default: SYSAUX Temporary TEMP
    [5] Additional Feature: Database Actions
    [6] Configure and start ORDS in Standalone Mode: Yes
    [7]    Protocol: HTTP
    [8]       HTTP Port: 8080
    [A] Accept and Continue - Create configuration and Install ORDS in the database
    [Q] Quit - Do not proceed. No changes
  Choose [A]:   <<<<<<<<<<<<< MY ENTRY <<<<<<<<<<<<<<<<
  .....
2024-07-11T12:23:05.339Z INFO        Oracle REST Data Services initialized
Oracle REST Data Services version : 24.2.2.r1871943
Oracle REST Data Services server info: jetty/10.0.21
Oracle REST Data Services java info: Java HotSpot(TM) 64-Bit Server VM 11.0.23+7-LTS-222

<<<< ORDS is running in foreground. Press ^C to stop it once installed 
^C


bash-4.4$

Once installed stop the ords server to configure it for mongodb:

# in the container as oracl user
bash-4.4$ /opt/oracle/ords/ords_pkg/bin/ords --config /opt/oracle/ords config set mongo.enabled true

ORDS: Release 24.2 Production on Thu Jul 11 12:25:09 2024

Copyright (c) 2010, 2024, Oracle.

Configuration:
  /opt/oracle/ords

The global setting named: mongo.enabled was set to: true

# start the ords server 
bash-4.4$ /opt/oracle/ords/ords_pkg/bin/ords --config /opt/oracle/ords serve

ORDS: Release 24.2 Production on Thu Jul 11 12:25:51 2024

Copyright (c) 2010, 2024, Oracle.

Configuration:
  /opt/oracle/ords

2024-07-11T12:25:53.083Z INFO        HTTP and HTTP/2 cleartext listening on host: 0.0.0.0 port: 8080
....
2024-07-11T12:25:53.170Z INFO        The Oracle API for MongoDB connection string is:
         mongodb://[{user}:{password}@]localhost:27017/{user}?authMechanism=PLAIN&authSource=$external&ssl=true&retryWrites=false&loadBalanced=true
2024-07-11T12:25:58.362Z INFO        Configuration properties for: |default|lo|
awt.toolkit=sun.awt.X11.XToolkit
db.tnsAliasName=FREEPDB1
...

If the configuration is correct the mongo shell URI is printed: 2024-07-11T12:25:53.170Z INFO The Oracle API for MongoDB connection string is:
mongodb://[{user}:{password}@]localhost:27017/{user}?

Connect as scott to the PDB and enable the schema for ORDS usage:

oracle@db23ai ~]$ sqlplus scott/tiger@localhost:1700/freepdb1

-- Enable SODA_APP role to work with JSON collections.
SQL> grant soda_app to scott;

SQL> exec ORDS.ENABLE_SCHEMA;

PL/SQL procedure successfully completed.

Now connect to Oracle database using mongosh:

[oracle@db23ai ~]$ export URI='mongodb://scott:tiger@localhost:27017/freepdb1?authMechanism=PLAIN&authSource=$external&tls=true&retryWrites=false&loadBalanced=true'

[oracle@db23ai ~]$ mongosh  --tlsAllowInvalidCertificates $URI
Current Mongosh Log ID:	668fd0b81156f2c04b482f8a
Connecting to:		mongodb://<credentials>@localhost:27017/freepdb1?authMechanism=PLAIN&authSource=%24external&tls=true&retryWrites=false&loadBalanced=true&serverSelectionTimeoutMS=2000&tlsAllowInvalidCertificates=true&appName=mongosh+2.2.12
Using MongoDB:		4.2.14
Using Mongosh:		2.2.12

For mongosh info see: https://docs.mongodb.com/mongodb-shell/


To help improve our products, anonymous usage data is collected and sent to MongoDB periodically (https://www.mongodb.com/legal/privacy-policy).
You can opt-out by running the disableTelemetry() command.

freepdb1> 

Let’s create a collection:

freepdb1> use scott

scott> db.createCollection('emp_mdb');

scott> db.emp_mdb.insertOne({"name":"Miller","job": "Programmer","salary": 70000});
{
  acknowledged: true,
  insertedId: ObjectId('668fd349ec28de3a61482f8d')
}

scott> db.emp_mdb.find({"name":"Miller"});
[
  {
    _id: ObjectId('668fd349ec28de3a61482f8d'),
    name: 'Miller',
    job: 'Programmer',
    salary: 70000
  }
]

# remark the table name in lowercase 
scott> show collections
emp_mdb

Let’s check the collection table from Oracle sqlplus point of view:

# connect to the PDB 
[oracle@db23ai ~]$ sqlplus scott/tiger@localhost:1700/freepdb1

SQL> 
-- describe the MongoDB collection stored as a table 
SQL> desc emp_mdb;
 Name					   Null?    Type
 ----------------------------------------- -------- ----------------------------
 DATA						    JSON

-- select the collection 
SQL> select * from emp_mdb;

DATA
--------------------------------------------------------------------------------
{"_id":"668fd349ec28de3a61482f8d","name":"Miller","job":"Programmer","salary":70


-- check the tables from scott schema (the collection is in lowercase ;) ) 
SQL> select table_name from user_tables;

TABLE_NAME
--------------------------------------------------------------------------------
DEPT
EMP
BONUS
SALGRADE
emp_mdb


-- print JSON fields a human readable 
SQL> select json_serialize(dbms_json_schema.describe(object_name=>'emp_mdb', owner_name  => 'SCOTT') pretty) as json_schema;

JSON_SCHEMA
--------------------------------------------------------------------------------
{
  "title" : "emp_mdb",
  "dbObject" : "SCOTT.emp_mdb",
  "type" : "object",
  "dbObjectType" : "table",
  "properties" :
  {
    "DATA" :
    {
    }
  },
  "dbPrimaryKey" :
  [
    "RESID"
  ]
}

--  print them more pretty 
SQL> select json_serialize((e.data) pretty) from emp_mdb e;

JSON_SERIALIZE((E.DATA)PRETTY)
--------------------------------------------------------------------------------
{
  "_id" : "668fd349ec28de3a61482f8d",
  "name" : "Miller",
  "job" : "Programmer",
  "salary" : 70000
}

Conclusion

Integrating MongoDB documents into an Oracle database enables organizations to take advantage of the unique strengths of both databases. This hybrid approach supports flexible, scalable, and robust data management, catering to a wide range of business needs. By following the steps outlined in this blog, you can create a seamless integration that leverages the best of both worlds.

Embrace the hybrid model and unlock new potentials in your data architecture.

L’article Integrating MongoDB Documents in Oracle Database: A Modern Approach to Hybrid Data Management est apparu en premier sur dbi Blog.

High-Level Summaries: The attention.log focuses on summarizing critical and significant events rather than logging every minor detail, including database startups and shutdowns, major configuration changes, errors or warnings that need immediate attention.

Consolidation of Critical Events: It provides a consolidated view of the most important events, making it easier for database administrators to quickly review and identify critical issues without rummaging through through detailed logs.

Accessibility: Designed to be easily readable and quickly accessible for a high-level overview of the database’s health and significant activities.

Complementary to Alert Log: While the attention.log highlights major events, it complements the alert.log rather than replacing it. Database administrators can use the attention.log for a quick overview and the alert.log for detailed diagnostics.

Location: Like the alert.log, the attention.log is also found in the DIAGNOSTIC_DEST directory, usually under $ORACLE_BASE/diag/rdbms/<db_name>/<instance_name>/log.

It can be very helpful for not so experienced database administrators or to get a quick overview in difficult or unexpected cases, as I had on a productive environment some time ago: an internal error had occurred, the database crashed and the alert.log was far too large to read without splitting it up which of course makes troubleshooting unnecessarily difficult (under time pressure).

How to get information about the attention.log:

The location of the attention log can be found by querying the V$DIAG_INFO view, it is in the same directory as the alert. log, (since Oracle 11g: $OH/diag/rdbms… )

select name, value
from   v$diag_info
where  name = 'Attention Log';

NAME                      VALUE
--------------------- -------------------------------------------------------------
Attention Log         /u01/app/oracle/diag/rdbms/cdb1/cdb1/trace/attention_cdb1.log

The oracle documentation proposes to query the V$DIAG_ALERT_EXT view to get relevant attention.log information, but it is a view over the XML-based alert.log (in the Automatic Diagnostic Repository for the current container), not the attention log! But nevertheless we can get very useful information out of it, divided into the same categories as in the attention.log:

--message_type 2=INCIDENT_ERROR, message_type 3=ERROR
SELECT message_type, message_level, message_text
FROM V$DIAG_ALERT_EXT 
WHERE message_type in (2, 3);

MESSAGE_TYPE MESSAGE_LEVEL MESSAGE_TEXT
------------ ------------- ---------------------------------------------------------
           3    4294967295 PMON (ospid: 3565): terminating the instance due to ORA error 471 

Querying V$DIAG_ALERT_EXT the most important labels are:

MESSAGE_LEVEL:

1: CRITICAL: critical errors

2: SEVERE: severe errors

8: IMPORTANT: important message

16: NORMAL: normal message

MESSAGE_TYPE:

1: UNKNOWN: essentially the NULL type

2: INCIDENT_ERROR: the program has encountered an error for some internal or unexpected reason, and it must be reported to Oracle Support

3: ERROR: an error of some kind has occurred

4: WARNING: an action occurred or a condition was discovered that should be reviewed and may require action

5: NOTIFICATION: reports a normal action or event, this could be a user action such as “logon completed”

6: TRACE: output of a diagnostic trace

Opening the attention.log with vi

vi /u01/app/oracle/diag/rdbms/cdb1/cdb1/trace/attention_cdb1.log

will give you an output like this (JSON formatted), which is obviously pretty comfortable to read:

{
IMMEDIATE : "PMON (ospid: 3565): terminating the instance due to ORA error 471" 
CAUSE: "PMON detected fatal background process death"
ACTION: "Termination of fatal background is not recommended, Investigate cause of process termination"
CLASS : CDB-INSTANCE / CDB_ADMIN / ERROR / DBAL-35782660
TIME : 2024-07-10T14:15:16.159-07:00
INFO : "Some additional data on error PMON error"
}

It is possible to convert the output into plain text format:

jq -r '.tags[].name' input.json > output.txt

which gives us a formatted expression that might look like this:

2024-07-01T10:15:32.123456+00:00
[SEVERE] ORA-00600: internal error code, arguments: [1234], [], [], [], [], [], [], [], [], [], []
Action: Please contact Oracle Support Services.

2024-07-01T11:20:45.789012+00:00
[CRITICAL] ORA-01578: ORACLE data block corrupted (file # 23, block # 220734)
Action: This error signifies a corrupted data block. The data block has been marked as corrupt. Consider restoring from backup.

2024-07-02T08:42:27.654321+00:00
[ALERT] Database instance crashed due to unexpected termination.
Action: Investigate the cause of the instance termination. Review related logs and diagnostic information.

2024-07-03T12:34:56.987654+00:00
[INFO] System global area (SGA) resized. New size: 68GB.
Action: No immediate action required. Monitor performance and stability.

Key Points

Severity Levels: Entries are tagged with severity levels such as [SEVERE], [CRITICAL], [IMPORTANT] or [NORMAL] to highlight their importance.

Timestamp: Each entry begins with a timestamp in ISO 8601 format.

Messages and Actions: Each entry provides a brief description of the event and recommended actions.

Benefits for DBAs

Quick Identification: The attention.log helps DBAs quickly identify and respond to critical issues without sifting through the more detailed alert.log.

Conciseness: It captures only the most significant events, reducing noise and making it easier to focus on urgent matters.

Complementary to alert.log: It complements the alert.log by summarizing critical events, while the alert.log continues to provide detailed information for troubleshooting.

Overall, the attention.log is a useful addition for DBAs, enabling more efficient monitoring and quicker responses to significant database events.

Sources, Links & Blogs:

https://oracle-base.com/articles/21c/attention-log-oracle-database-21c

https://blogs.oracle.com/cloud-infrastructure/post/alert-log-support-for-oci-database-management

https://docs.oracle.com/en/database/oracle/oracle-database/21/nfcon/management-solutions.html#GUID-F2EB58EC-4B22-473F-A2D3-40161372610E

L’article Oracle 21c: Attention Log – Useful or Superflous? est apparu en premier sur dbi Blog.

In our data-centric world, efficiently searching and retrieving information is crucial. Traditional search methods frequently fail with the surge of unstructured data. Oracle 23c AI addresses this challenge with a groundbreaking feature: vector search. This innovative tool transforms data handling and retrieval, particularly with complex, high-dimensional datasets. In this blog post, we’ll explore the concept of vector search, its implementation in Oracle 23c AI, and its transformative impact across various industries.

Understanding Vector Search (short)

Vector search, also known as similarity search or nearest neighbor search, involves searching for data points in a vector space. Unlike traditional search methods that rely on exact keyword matches, vector search leverages mathematical representations of data (vectors) to find similarities. Each data point is encoded into a vector, capturing its essential features. The search process then identifies vectors that are closest to the query vector based on a chosen distance metric, such as Euclidean distance or cosine similarity.

Oracle 23c AI integrates vector search capabilities, providing users with an advanced tool for data retrieval. Here’s a closer look at how it works and its features:

Seamless Integration: Oracle 23c AI’s vector search is integrated into the database, allowing users to perform vector searches without the need for external tools or complex workflows.

High Performance: Leveraging Oracle’s robust infrastructure, vector search in 23c AI offers high-speed search capabilities even with large datasets. Advanced indexing and optimized algorithms ensure quick and accurate retrieval.

Multi-Modal Data Support: Oracle 23c AI supports various data types, including text, images, audio, and more. This versatility makes it a powerful tool for applications across different domains.

Customizable Distance Metrics: Users can choose from a variety of distance metrics based on their specific needs, enhancing the flexibility and accuracy of the search results.

What is the flow

Data Ingestion: Data is ingested into the Oracle 23c AI database, where it is preprocessed and converted into vectors. This step involves using machine learning models to encode the data’s features into numerical vectors.

Indexing: The vectors are indexed to facilitate efficient searching. Oracle 23c AI uses advanced indexing techniques, such as hierarchical navigable small world (HNSW) graphs, to enable fast and scalable searches.

Querying: When a query is made, it is also converted into a vector. The vector search algorithm then identifies the closest vectors in the database using the chosen distance metric.

Results: The search results are returned, showcasing the most similar data points to the query. These results can be further refined or analyzed based on the application’s requirements.

The classical example is the image search, and classification following some criteria.

To achieve this goal Oracle can make benefit of ONNX (Open Neural Network Exchange) existent models. ONNX models offer a standardised format for representing deep learning models.

Predefined ONNX models exist for free: https://github.com/onnx/models

Building the test environment

I use an OCI VM Oracle Linux Server 9.4

I will use Python function to generate the ONNX model.

First step is to create a virtual environnement using python:

(venv) [oracle@db23-134956 ~]$ pip freeze
certifi==2024.6.2
cffi==1.16.0
charset-normalizer==3.3.2
coloredlogs==15.0.1
contourpy==1.2.1
cryptography==42.0.8
cycler==0.12.1
filelock==3.15.4
flatbuffers==24.3.25
fonttools==4.53.0
fsspec==2024.6.1
huggingface-hub==0.23.4
humanfriendly==10.0
idna==3.7
importlib_resources==6.4.0
Jinja2==3.1.4
joblib==1.4.2
kiwisolver==1.4.5
MarkupSafe==2.1.5
matplotlib==3.9.0
mpmath==1.3.0
networkx==3.3
numpy==1.26.4
nvidia-cublas-cu12==12.1.3.1
nvidia-cuda-cupti-cu12==12.1.105
nvidia-cuda-nvrtc-cu12==12.1.105
nvidia-cuda-runtime-cu12==12.1.105
nvidia-cudnn-cu12==8.9.2.26
nvidia-cufft-cu12==11.0.2.54
nvidia-curand-cu12==10.3.2.106
nvidia-cusolver-cu12==11.4.5.107
nvidia-cusparse-cu12==12.1.0.106
nvidia-nccl-cu12==2.20.5
nvidia-nvjitlink-cu12==12.5.82
nvidia-nvtx-cu12==12.1.105
oml @ file:///home/oracle/installer/client/oml-2.0-cp312-cp312-linux_x86_64.whl#sha256=0c1f7c83256f60c87f1f66b2894098bc8fefd8a60a03e67ba873661dd178b3c2
onnx==1.16.1
onnxruntime==1.18.1
onnxruntime_extensions==0.11.0
oracledb==2.2.1
packaging==24.1
pandas==2.2.2
pillow==10.4.0
protobuf==5.27.2
pycparser==2.22
pyparsing==3.1.2
python-dateutil==2.9.0.post0
pytz==2024.1
PyYAML==6.0.1
regex==2024.5.15
requests==2.32.3
safetensors==0.4.3
scikit-learn==1.5.0
scipy==1.13.1
sentencepiece==0.2.0
setuptools==70.2.0
six==1.16.0
sympy==1.12.1
threadpoolctl==3.5.0
tokenizers==0.19.1
torch==2.3.1
tqdm==4.66.4
transformers==4.42.3
typing_extensions==4.12.2
tzdata==2024.1
urllib3==2.2.2
zipp==3.19.2

I also use Oracle Database 23ai Free as docker image.

To start the docker image I use this command, and I put datafiles and archivelogs on external mount point:

(venv) [oracle@db23-134956 ~]$ docker run  -it --name 23ai -p 1700:1521 \
-v /u02/data/DB:/opt/oracle/oradata \
-v /u02/reco/DB:/opt/oracle/reco \
-e ENABLE_ARCHIVELOG=true  \
container-registry.oracle.com/database/free

Let’s get the ONX image and built it for our usage:

(venv) [oracle@db23-134956 ~]$ python
Python 3.12.1 (main, Feb 19 2024, 00:00:00) [GCC 11.4.1 20231218 (Red Hat 11.4.1-3.0.1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from oml.utils import EmbeddingModel, EmbeddingModelConfig
>>> em = EmbeddingModel(model_name="sentence-transformers/all-MiniLM-L6-v2")
>>> em.export2file("all-MiniLM-L6-v2", output_dir=".")
tokenizer_config.json: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 350/350 [00:00<00:00, 4.01MB/s]
vocab.txt: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 232k/232k [00:00<00:00, 1.15MB/s]
special_tokens_map.json: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 112/112 [00:00<00:00, 1.52MB/s]
tokenizer.json: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 466k/466k [00:00<00:00, 1.55MB/s]
config.json: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 612/612 [00:00<00:00, 8.23MB/s]
model.safetensors: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 90.9M/90.9M [00:00<00:00, 304MB/s]
>>> exit()
l(venv) [oracle@db23-134956 ~]$ ls
all-MiniLM-L6-v2.onnx  venv

The ONNX generated file is all-MiniLM-L6-v2.onnx

Finally I copy the ONNX generated file to a shared directory to be able to import it to the database:

(venv) [oracle@db23-134956 ~]$ cp all-MiniLM-L6-v2.onnx /u02/data/DB/FREE/FREEPDB1/dump/1AC27018093909F8E063020011AC7FF6/

Let’s connect to the database and load the ONNX model into database:

(venv) [oracle@db23-134956 ~]$ sqlplus system/oracle@localhost:1700/freepdb1

SQL*Plus: Release 23.0.0.0.0 - Production on Wed Jul 3 09:12:02 2024
Version 23.4.0.24.05

SQL> create or replace directory DATA_PUMP_DIR as '/opt/oracle/oradata/FREE/FREEPDB1/dump';

Directory created.

SQL> begin
  2  dbms_vector.load_onnx_model(
  3  directory => 'DATA_PUMP_DIR'
  4  , file_name => 'all-MiniLM-L6-v2.onnx'
  5  , model_name => 'all_minilm_l6_v2'
  6  , metadata => json('{"function" : "embedding", "embeddingOutput" : "embedding" , "input": {"input": ["DATA"]}}')
  7  );
  8  end;
  9  /

PL/SQL procedure successfully completed.

-- test the loaded model 
SQL> select
  model_name
  , mining_function
  , algorithm
  , (model_size/1024/1024) as model_size_mb
from user_mining_models
order by model_name;  2    3    4    5    6    7

MODEL_NAME
--------------------------------------------------------------------------------
MINING_FUNCTION 	       ALGORITHM		      MODEL_SIZE_MB
------------------------------ ------------------------------ -------------
ALL_MINILM_L6_V2
EMBEDDING		       ONNX				 86.4376068

The test table contains a word, the description end the generated vector:

-- create the table  
SQL> create table my_dictionary(
 id number generated always as identity
  , word varchar2(100) not null
  , description varchar2(500) not null
  , word_vector vector not null
   );

Table created.

-- create the trigger to generate the vector 
SQL> create or replace trigger my_dict_vect_build
before insert or update on my_dictionary
for each row
declare
begin
  :new.word_vector := dbms_vector_chain.utl_to_embedding(
    data => :new.word
    , params => json(q'[{"provider": "database",  2   "model": "&model_name."}]')
  );
end;
/  3    4    5    6    7    8    9   10   11
Enter value for model_name: all_minilm_l6_v2
old   8:     , params => json(q'[{"provider": "database",  2   "model": "&model_name."}]')
new   8:     , params => json(q'[{"provider": "database",  2   "model": "all_minilm_l6_v2"}]')

Trigger created.

-- create and index on the table
SQL> create vector index my_dict_ivf_idx
  on my_dictionary(word_vector)
  organization neighbor partitions
  distance cosine
  with target accuracy 95;

Index created.

I download a dictionary from the web in CSV mode as word | description from https://www.bragitoff.com/2016/03/english-dictionary-in-csv-format and I formatted it to be somehow like this :

Score;q'[To engrave, as upon a shield.]'
Score;q'[To make a score of, as points, runs, etc., in a game.]'
Score;q'[To write down in proper order and arrangement  as, to score an overture for an orchestra. See Score, n., 9.]'
Score;q'[To mark with parallel lines or scratches  as, the rocks of New England and the Western States were scored in the drift epoch.]'
Scorer;q'[One who, or that which, scores.]'

The I loaded the dictionary into a table dict using sqlldr:

SQL> 

SQL> desc dict;
Name Null? Type

WORD VARCHAR2(100)
DESCRIPTION NOT NULL VARCHAR2(500)

Using these parameters files: 

[oracle@0c97e3a7be16 ]$ cat ld.par
control=ld.ctl
log=ld.log
bad=ld.bad
data=ld.csv
direct=true
errors=1000000

[oracle@0c97e3a7be16 ]$ cat ld.ctl 
load data into table dict
insert
fields terminated by ";"
(word,description
)

[oracle@0c97e3a7be16 1AC27018093909F8E063020011AC7FF6]$  sqlldr 'system/"Hello-World-123"@freepdb1' parfile=ld.par

SQL*Loader: Release 23.0.0.0.0 - Production on Tue Jul 9 09:47:59 2024
Version 23.4.0.24.05

Copyright (c) 1982, 2024, Oracle and/or its affiliates.  All rights reserved.

Path used:      Direct

Load completed - logical record count 54555.

Table DICT:
  19080 Rows successfully loaded.

Check the log file:
  ld.log
for more information about the load.

SQL> select count(*) from dict;

  COUNT(*)
----------
     19080

Finally insert into the final table my_dictionars. The trigger will build the vector information

SQL> insert into my_dictionary(word,DESCRIPTION)  select WORD,DESCRIPTION from dict;

19080 rows created.

SQL> commit;

Commit complete.

Let’s test the miraculous AI.

The ONNX model used was designed to search words. Obviously more the model is complicates mode the research is precise, and more is expensive.

The question is what are the first x words from my build dictionary which are most closely to a given word. As My dictionary contains words similar I used distinct to filter the final result.

Let’s search the result for the word “Sun”:

SQL> define search_term="Sun";

select distinct word from (
with subject as (
  select to_vector(
  vector_embedding(&model_name. using '&search_term.' as data)
  ) as search_vector
  )
  select o.word from my_dictionary o, subject s order by cosine_distance(
    o.word_vector, s.search_vector)
  fetch approx first 50 rows only with target accuracy 80);  2    3    4    5    6    7    8    9
old   4:   vector_embedding(&model_name. using '&search_term.' as data)
new   4:   vector_embedding(all_minilm_l6_v2 using 'Sun' as data)

WORD
--------------------------------------------------------------------------------
Radiant
Radiate
Light
Radiancy
Earth shine
Heat

6 rows selected.

Test for the word “Seed”:

SQL> define search_term="Seed";
SQL> /
old   4:   vector_embedding(&model_name. using '&search_term.' as data)
new   4:   vector_embedding(all_minilm_l6_v2 using 'Seed' as data)

WORD
--------------------------------------------------------------------------------
Germinate
Germination
Feeder
Grow
Germinal
Heartseed
Grass
Fertile
Fertilization
Grass-grown
Harvest
Fruit
Grower
Packfong
Pack
Agnation
Gusset

17 rows selected.

The result’s is pretty cool

Conclusion

The AI is here, everyone is using it. We are going to use it more and more. Artificial Intelligence (AI) has rapidly evolved from a theoretical concept into a transformative force across numerous industries, redefining how we interact with technology and fundamentally altering societal structures.

This example is limited, the ONNX model is simple, but shows how easy we can implemented a solution which can project into the AI world.

L’article Exploring Vector Search in Oracle 23c AI est apparu en premier sur dbi Blog.

Desupport of Traditional Auditing: Traditional auditing is desupported in Oracle 23ai, which means all auditing must now be done using unified auditing. Unified auditing consolidates audit records into a single, secure trail and supports conditional policies for selective auditing.

Column-Level Audit Policies: One of the most notable features is the ability to create audit policies on individual columns of tables and views. This granularity control allows organizations to focus their audit efforts on the most sensitive data elements, reducing the volume of audit data and enhancing security by targeting specific actions like SELECT, INSERT, UPDATE, or DELETE on specified columns​. It can be implemented like:

conn / as sysdba
noaudit policy test_audit_policy;
drop audit policy test_audit_policy;
create audit policy test_audit_policy
	actions update(col1, col2) on userx.audit_test_tab,
	select(col2) on userx.audit_test_tab
	container = current;
audit policy test_audit_policy;

Enhanced Integration with IAM and Cloud Services: Oracle 23ai includes improved authentication and authorization capabilities for IAM users, supporting Azure AD OAuth2 tokens and allowing seamless integration with Oracle Autonomous Database on Dedicated Exadata Infrastructure.

New Cryptographic Algorithms and Security Enhancements: The release includes support for new cryptographic algorithms such as SM2, SM3, SM4, and SHA-3, along with enhancements to the DBMS_CRYPTO package and orapki utility.

Consolidation of FIPS 140 Parameter: Oracle 23c introduces a unified FIPS_140 parameter for configuring FIPS across various database environments, including Transparent Data Encryption (TDE) and network encryption, streamlining the compliance process and improving security management​.

Cleanup of audit trail

Also for the reason that unified audit is now standard, I would like to summarize the management and cleanup of the audit trail in detail here:

First of all you should check the size of the existing audit tables (AUD$ and FGA_LOG$), create a new audit-tablespace for the audit tables:

set lin 999
set pages 999
col tablespace_name for a30
col segment_name for a30
col owner for a20
select owner, segment_name, tablespace_name, bytes/1024/1024 MB 
from dba_segments 
where segment_name in ('AUD$', 'FGA_LOG$');
-- create audit tablespace
create tablespace audit_ts;
select file_name, file_id, tablespace_name, autoextensible, maxbytes/1024/1024/1024 GB 
from dba_data_files 
where tablespace_name like 'AUD$';
-- alter database datafile … autoextend on next 100M maxsize 1G;
-- eventually create an audit history table for further use
create table AUD_HIST tablespace audit_ts as select * from AUD$;
-- truncate table sys.aud$;

Now you can move the audit tables from sysaux to the audit-tablespace:

--move AUD$
BEGIN
DBMS_AUDIT_MGMT.SET_AUDIT_TRAIL_LOCATION(
audit_trail_type => DBMS_AUDIT_MGMT.AUDIT_TRAIL_AUD_STD,
audit_trail_location_value => 'AUDIT_TS');
END;
/
--move FGA_LOG$
BEGIN
DBMS_AUDIT_MGMT.SET_AUDIT_TRAIL_LOCATION(
audit_trail_type => DBMS_AUDIT_MGMT.AUDIT_TRAIL_FGA_STD,
audit_trail_location_value => 'AUDIT_TS');
END;
/
--moving additional audit tables
alter table FGACOL$ move tablespace AUDIT_TS;

Short check of audit-parameters:

-- check audit parameters
col parameter_name for a30
col parameter_value for a20
col audit_trail for a20
show parameter audit
SELECT * FROM dba_audit_mgmt_config_params;

Before starting any cleanup you should be sure that no old jobs exist:

-- deinit all purge jobs
exec dbms_audit_mgmt.deinit_cleanup( dbms_audit_mgmt.audit_trail_all );
exec dbms_audit_mgmt.deinit_cleanup( dbms_audit_mgmt.audit_trail_os );
exec dbms_audit_mgmt.deinit_cleanup( dbms_audit_mgmt.audit_trail_xml );
exec dbms_audit_mgmt.deinit_cleanup( dbms_audit_mgmt.audit_trail_aud_std );
exec dbms_audit_mgmt.deinit_cleanup( dbms_audit_mgmt.audit_trail_fga_std )

Now you can set up an audit-management, e.g. every 24 hours, which purges audit-entries older than 7 months:

-- init cleanup for 24 hours
exec DBMS_AUDIT_MGMT.INIT_CLEANUP( DBMS_AUDIT_MGMT.AUDIT_TRAIL_all, 24 );
-- set retention (last_archive_timestamp, sysdate -210 days) =7 months
exec dbms_audit_mgmt.set_last_archive_timestamp( dbms_audit_mgmt.AUDIT_TRAIL_AUD_STD, to_timestamp(sysdate-210));
exec dbms_audit_mgmt.set_last_archive_timestamp( dbms_audit_mgmt.AUDIT_TRAIL_FGA_STD, to_timestamp(sysdate-210));
exec DBMS_AUDIT_MGMT.SET_LAST_ARCHIVE_TIMESTAMP( dbms_audit_mgmt.audit_trail_unified, sysdate-210);
-- 1 means RAC-node 1
exec dbms_audit_mgmt.set_last_archive_timestamp( dbms_audit_mgmt.AUDIT_TRAIL_OS, to_timestamp(sysdate-210), 1);
exec dbms_audit_mgmt.set_last_archive_timestamp( dbms_audit_mgmt.AUDIT_TRAIL_XML, to_timestamp(sysdate-210), 1);

Short check of parameter settings:

-- getting parameters
COLUMN parameter_name FORMAT A30
COLUMN parameter_value FORMAT A20
COLUMN audit_trail FORMAT A20
col days_back  for 9999  head "days|back"
select audit_trail, rac_instance, last_archive_ts, extract(day from(systimestamp-last_archive_ts))days_back from DBA_AUDIT_MGMT_LAST_ARCH_TS;

Now you can set up and initiate the purge-jobs:

-- creating purge job for all audit trails
BEGIN 
dbms_audit_mgmt.create_purge_job(
audit_trail_type => dbms_audit_mgmt.audit_trail_all,
audit_trail_purge_interval => 24  /* hours */,
audit_trail_purge_name => 'DAILY_AUDIT_PURGE',
use_last_arch_timestamp => TRUE);
END;
/
-- create job for automatic updating last_archive_timestamp
create or replace procedure AUDIT_UPDATE_RETENTION( m_purge_retention  IN  number  DEFAULT 210
) AS
BEGIN
DBMS_AUDIT_MGMT.SET_LAST_ARCHIVE_TIMESTAMP( DBMS_AUDIT_MGMT.AUDIT_TRAIL_AUD_STD, TO_TIMESTAMP(SYSDATE-m_purge_retention));
DBMS_AUDIT_MGMT.SET_LAST_ARCHIVE_TIMESTAMP( DBMS_AUDIT_MGMT.AUDIT_TRAIL_AUD_STD, TO_TIMESTAMP(SYSDATE-m_purge_retention));
DBMS_AUDIT_MGMT.SET_LAST_ARCHIVE_TIMESTAMP( DBMS_AUDIT_MGMT.AUDIT_TRAIL_FGA_STD, TO_TIMESTAMP(SYSDATE-m_purge_retention));
DBMS_AUDIT_MGMT.SET_LAST_ARCHIVE_TIMESTAMP( DBMS_AUDIT_MGMT.AUDIT_TRAIL_UNIFIED, TO_TIMESTAMP(SYSDATE-m_purge_retention));
DBMS_AUDIT_MGMT.SET_LAST_ARCHIVE_TIMESTAMP( DBMS_AUDIT_MGMT.AUDIT_TRAIL_OS, TO_TIMESTAMP(SYSDATE-m_purge_retention), 1);
DBMS_AUDIT_MGMT.SET_LAST_ARCHIVE_TIMESTAMP( DBMS_AUDIT_MGMT.AUDIT_TRAIL_XML, TO_TIMESTAMP(SYSDATE-m_purge_retention), 1);
END;
/

You can test it now:

-- test run
exec audit_update_retention(210);

Before we invoke the new job, we get sure that no old bodies are left:

-- drop job if exists
set serveroutput on
begin
dbms_scheduler.drop_job(job_name => 'DAILY_AUDIT_UPDATE_RETENTION');
end;
/
-- create job for last_archive_timestamp
set serveroutput on
declare
m_purge_retention number DEFAULT 210;
ex_job_doesnt_exist EXCEPTION;
PRAGMA EXCEPTION_INIT(ex_job_doesnt_exist, -27475);
begin
--dbms_scheduler.drop_job(job_name => 'DAILY_AUDIT_UPDATE_RETENTION');
--dbms_output.put_line(chr(13)||'= 60;
DBMS_SCHEDULER.create_job (
job_name => 'DAILY_AUDIT_UPDATE_RETENTION',
job_type => 'PLSQL_BLOCK',
job_action => 'BEGIN
DBMS_AUDIT_MGMT.SET_LAST_ARCHIVE_TIMESTAMP( DBMS_AUDIT_MGMT.AUDIT_TRAIL_AUD_STD, TO_TIMESTAMP(SYSDATE-m_purge_retention));
DBMS_AUDIT_MGMT.SET_LAST_ARCHIVE_TIMESTAMP( DBMS_AUDIT_MGMT.AUDIT_TRAIL_AUD_STD, TO_TIMESTAMP(SYSDATE-m_purge_retention));
DBMS_AUDIT_MGMT.SET_LAST_ARCHIVE_TIMESTAMP( DBMS_AUDIT_MGMT.AUDIT_TRAIL_FGA_STD, TO_TIMESTAMP(SYSDATE-m_purge_retention));
DBMS_AUDIT_MGMT.SET_LAST_ARCHIVE_TIMESTAMP( DBMS_AUDIT_MGMT.AUDIT_TRAIL_UNIFIED, TO_TIMESTAMP(SYSDATE-m_purge_retention));
DBMS_AUDIT_MGMT.SET_LAST_ARCHIVE_TIMESTAMP( DBMS_AUDIT_MGMT.AUDIT_TRAIL_OS, TO_TIMESTAMP(SYSDATE-m_purge_retention), 1);
DBMS_AUDIT_MGMT.SET_LAST_ARCHIVE_TIMESTAMP( DBMS_AUDIT_MGMT.AUDIT_TRAIL_XML, TO_TIMESTAMP(SYSDATE-m_purge_retention), 1);
END;', 
start_date => SYSTIMESTAMP, 
repeat_interval => 'freq=daily; byhour=0; byminute=0; bysecond=0;', 
end_date => NULL, 
enabled => TRUE, 
auto_drop => FALSE, 
comments => 'every day, update last_archive_timestamp (which DAILY_AUDIT_PURGE uses) to '||m_purge_retention||' days back.'
);
--Exception
--when ex_must_be_declared then
--dbms_output.put_line('DAILY_AUDIT_UPDATE_RETENTION: component missing?'||chr(13));
END;
/

An other option would be to start the jon manually or to purge the audit-trail on demand:

-- manual start of purge jobs
select distinct job_name, owner from dba_scheduler_jobs;
exec dbms_scheduler.run_job('DAILY_AUDIT_UPDATE_RETENTION',use_current_session=>false);
exec dbms_scheduler.run_job('audsys.DAILY_AUDIT_PURGE',use_current_session=>false);
-- other options
exec DBMS_AUDIT_MGMT.FLUSH_UNIFIED_AUDIT_TRAIL;
-- or
begin
DBMS_AUDIT_MGMT.FLUSH_UNIFIED_AUDIT_TRAIL;
for i in 1..10 loop
DBMS_AUDIT_MGMT.TRANSFER_UNIFIED_AUDIT_RECORDS;
end loop;
end;
/
begin
DBMS_AUDIT_MGMT.CLEAN_AUDIT_TRAIL(
audit_trail_type => DBMS_AUDIT_MGMT.AUDIT_TRAIL_UNIFIED,
use_last_arch_timestamp => FALSE);
end;
/

Sources, links and related blogs:

https://support.oracle.com/knowledge/Oracle%20Database%20Products/2904294_1.html

https://docs.oracle.com/en/database/oracle/oracle-database/19/sqlrf/AUDIT-Unified-Auditing.html

https://oracle-base.com/articles/23/auditing-enhancements-23

L’article Cleanup Audit Trail – the Full Picture est apparu en premier sur dbi Blog.

Creating Immutable Backups Using RMAN

It is now possible to store immutable backups with the Oracle Database Cloud Backup Module for Oracle Cloud Infrastructure (OCI) which enables to configure backups suchlike, that they cannot be altered or deleted for a specific period, which helps to follow any compliance and data protection rules. Immutable backups will prevent anyone, even administrators to delete or modify backups in OCI Object Storage.

Here’s a step-by-step example to set up immutable backups using RMAN and the Oracle Database Cloud Backup Module for OCI:

You have to download and install the Oracle Database Cloud Backup Module on your database server whith following options: You can specify the -bucket parameter (and the name of an existing bucket or a new immutable bucket that you have created in OCI) otherwise the default bucket created by the installer will be used.

java -jar oci_install.jar -host https://objectstorage.<region>.oraclecloud.com -pdb1 <password> -opcId "<tenancy-namespace>/<bucket-name>" -opcPassFile <path-to-opc-pass-file> -libDir <path-to-lib-dir>

In RMAN you have to configure your SBT channel that way, that it will use the Oracle Database Cloud Backup Module. Now you can create a backup, e.g immutable for 30 days:

• KEEP UNTIL TIME ‘SYSDATE+30’: specifies that the backup is retained and immutable for 30 days.
• IMMUTABLE: marks the backup as immutable.

A backup-script (like following example) can be used:

RMAN> connect target /
RMAN> CONFIGURE CHANNEL DEVICE TYPE sbt PARMS 'SBT_LIBRARY=/opt/oracle/lib/libopc.so, SBT_PARMS=(OPC_PFILE=/opt/oracle/opc/ocipassfile)';
BACKUP AS BACKUPSET DATABASE KEEP UNTIL TIME 'SYSDATE+30' IMMUTABLE;
RMAN> LIST BACKUP OF DATABASE SUMMARY;

You should ensure,that the backup status is AVAILABLE und and the backupset includes the KEEP UNTIL date.

That OCI backups work appropiatley you have to take into consideration, that

• OCI Object Storage bucket has Object Lock enabled to support immutability.
• The IMMUTABLE keyword is used for specifying that the backup should not be altered or deleted within the specified retention.
• OCI Object Storage policy and IAM settings must allow creating immutable objects.

RMAN Backup Encryption Algorithm Now Defaults to AES256

From Oracle 23ai on RMAN encrypted backups default to AES256 encryption algorithm.

For backward compatibility existing backups created with previous encryption settings remain accessible. However, new backups will use AES256 unless explicitly configured otherwise. In other words, restore is supported by using existing backupsets AES128 or AES192 encryption algorithms and by changing your default settings you can still create backups with AES128 encryption.

With AES256 encryption you will get enhanced security using a stronger encryption standard and achieving industry standards and compliance regulations for sensitive data.

The default encryption using AES256 can be easily invoked out-of-the-box:

RMAN> SET ENCRYPTION ON IDENTIFIED BY 'your_password';
RMAN> BACKUP DATABASE;

Oracle Globally Distributed Database Coordinated Backup and Restore Enhancements

Additionally Oracle 23c AI Database includes several enhancements to globally distributed database coordinated backup and restore operations, aiming to improve efficiency, consistency, and reliability in multi-datacenter environments. Here are some key enhancements:

Unified Backup Management can simplify the management of backups across globally distributed databases by providing a centralized framework to coordinate backup operations. This ensures that backups are synchronized and consistent across all sites. It can be programmed via DBMS_BACKUP_RESTORE:

-- Example SQL script to configure unified backup policy
-- Define global backup policy 
BEGIN 
DBMS_BACKUP_RESTORE.SET_GLOBAL_POLICY( 
policy_name => 'GlobalBackupPolicy', 
backup_schedule => 'FULL EVERY SUNDAY 2:00 AM UTC, INCREMENTAL DAILY 2:00 AM UTC', 
retention_period => 30, 
encryption_algorithm => 'AES256' 
); 
END; 
/ 
-- Apply global backup policy to all distributed databases 
BEGIN 
DBMS_BACKUP_RESTORE.APPLY_GLOBAL_POLICY( 
policy_name => 'GlobalBackupPolicy', 
target_databases => 'NYC_DB, LDN_DB, TOKYO_DB' 
); 
END; 
/

Global Backup Policies: You can now define global backup policies that apply uniformly across all distributed databases. This standardization helps in maintaining consistency and compliance with organizational policies. It can be set up like:

BEGIN 
DBMS_BACKUP_RESTORE.CONFIGURE_GLOBAL_SETTING(
setting_name => 'RETENTION_POLICY', 
setting_value => 'RECOVERY WINDOW OF 30 DAYS'
 ); 
DBMS_BACKUP_RESTORE.CONFIGURE_GLOBAL_SETTING( 
setting_name => 'BACKUP_OPTIMIZATION', 
setting_value => 'ON' 
); 
DBMS_BACKUP_RESTORE.CONFIGURE_GLOBAL_SETTING( 
setting_name => 'ENCRYPTION_ALGORITHM', 
setting_value => 'AES256' 
); 
DBMS_BACKUP_RESTORE.CONFIGURE_GLOBAL_SETTING( 
setting_name => 'ENCRYPTION_PASSWORD', 
setting_value => 'your_encryption_password'
 ); 
END; 
/

Consistent Point-in-Time Recovery ensures that all databases in a distributed environment can be restored to the same point in time, maintaining data consistency across different geographic locations (in my opinion not really new, it is managed via known RMAN-scripts for restore & recover).

Cross-Site Transaction Coordination enhances the coordination of transactions across distributed databases to ensure that backup and restore operations capture a consistent state of the entire database system, it can be done via Oracle Global Data Services (GDS):

-- Connect to the GDS catalog database
sqlplus / as sysdba

-- Create the GDS catalog
BEGIN
  DBMS_GDS.CREATE_GDS_CATALOG();
END;
/

-- Add databases to the GDS pool
BEGIN
  DBMS_GDS.ADD_GDS_DATABASE(
    db_unique_name => 'NYC_DB',
    connect_string => 'NYC_DB_CONN_STRING',
    region         => 'AMERICAS'
  );

  DBMS_GDS.ADD_GDS_DATABASE(
    db_unique_name => 'LDN_DB',
    connect_string => 'LDN_DB_CONN_STRING',
    region         => 'EMEA'
  );

  DBMS_GDS.ADD_GDS_DATABASE(
    db_unique_name => 'TOKYO_DB',
    connect_string => 'TOKYO_DB_CONN_STRING',
    region         => 'APAC'
  );
END;
/
-- On each database (NYC_DB, LDN_DB, TOKYO_DB)
sqlplus / as sysdba

-- Enable distributed transactions
ALTER SYSTEM SET distributed_transactions = 10 SCOPE = BOTH;

-- Configure the global_names parameter
ALTER SYSTEM SET global_names = TRUE SCOPE = BOTH;

-- Set the commit point strength
ALTER SYSTEM SET commit_point_strength = <value> SCOPE = BOTH;

-- Configure Oracle Net for distributed transactions (update tnsnames.ora and listener.ora as necessary)
-- Create database links to enable cross-site communication
-- Begin the distributed transaction
SET TRANSACTION READ WRITE;

...

-- Commit the transaction
COMMIT;

Parallel Backup Streams support parallel backup streams to expedite the backup process, making it faster and more efficient, especially for large databases spread across multiple sites. You just have to configure RMAn-channels like in previous releases but in Oracle 23ai Adaptive Parallelism is used. By setting a higher level of parallelism (e.g., 8), Oracle 23ai can dynamically adjust the number of active channels based on real-time system performance and workload, rather than strictly adhering to the configured number.

Automated Restore Coordination automates the coordination of restore operations across multiple databases, ensuring that all parts of the distributed database are restored in a synchronized manner, it will be invoked with:

-- Enable automated restore coordination
CONFIGURE RESTORE COORDINATION ON;
-- Automated restore coordination across distributed databases
RESTORE DATABASE FROM SERVICE 'NYC_DB' USING CHANNEL c1;
RESTORE DATABASE FROM SERVICE 'LDN_DB' USING CHANNEL c2;
RESTORE DATABASE FROM SERVICE 'TOKYO_DB' USING CHANNEL c3;
-- Recover database
RECOVER DATABASE;

Resilient Backup Infrastructure enhancements to the backup infrastructure to handle network disruptions and other issues that may arise in a globally distributed environment, ensuring that backups are resilient and reliable. Oracle Database 23ai enhances the resilience of backup infrastructure by integrating features like Data Guard for high availability, Fast Recovery Area for automated backup management, and RMAN duplication for redundant data copies.

RMAN Operational Enhancements

Automatic Block Repair During Backup: Oracle 23ai enhances RMAN’s capability to automatically detect and repair corrupt blocks during backup operations.

RMAN> BACKUP DATABASE PLUS ARCHIVELOG CHECK READONLY;

Improved Block Corruption Detection: Oracle 23ai has enhanced algorithms for faster and more accurate detection of block corruption during backup and restore operations and the some structures are used for Enhanced Backup Validation: RMAN has now faster and more efficient methods to validate backups and ensure data integrity.

Simplified Database Migration Across Platforms Using RMAN

Oracle Database 23ai includes RMAN-enhancements which simplify the process of migrating databases across different platforms. This feature is particularly useful when transitioning databases between heterogeneous environments, ensuring minimal downtime and efficient migration. New command options allow existing RMAN backups to be used to transport tablespaces or pluggable databases to a new destination database with minimal downtime. With the example below you can easily migrate from e.g. Linux to Windows or vice versa:

-- Connect RMAN to both source and target databases
CONNECT TARGET sys@PROD_DB
CONNECT AUXILIARY sys@TARGET_DB

-- Configure cross-platform migration settings
SET NEWNAME FOR DATABASE TO '/path/to/new/PROD_DB';

-- Start migration process
DUPLICATE TARGET DATABASE TO TARGET_DB
  FROM ACTIVE DATABASE
  SPFILE
  PARAMETER_VALUE_CONVERT 'db_unique_name=PROD_DB','db_unique_name=TARGET_DB'
  SET db_file_name_convert='/prod_data/','/target_data/'
  SET log_file_name_convert='/prod_redo/','/target_redo/';
-- Check the migration progress
SHOW DUPLICATE SUMMARY;
-- Validate the migrated database
VALIDATE DATABASE;

Related Sources, Links and Blogs:

https://docs.oracle.com/en/database/oracle/oracle-database/23/nfcoa/oracle-database-23c-new-features-guide.pdf

https://docs.oracle.com/en/cloud/paas/db-backup-cloud/csdbb/storing-backups-oci-immutable-buckets.html#GUID-DECFAAF9-861F-46D9-A1FC-B848476772C5

https://www.oracle.com/pls/topic/lookup?ctx=en/database/oracle/oracle-database/23&id=SHARD-GUID-99A32370-00BD-4C30-A2DB-19F4EA168064

https://www.oracle.com/pls/topic/lookup?ctx=en/database/oracle/oracle-database/23&id=BRADV-GUID-6ED708C7-1092-45FC-80C6-236F062D0DAC

https://www.oracle.com/pls/topic/lookup?ctx=en/database/oracle/oracle-database/23&id=BRADV-GUID-E836E243-6620-495B-ACFB-AC0001EF4E89

https://www.oracle.com/pls/topic/lookup?ctx=en/database/oracle/oracle-database/23&id=BRADV-GUID-BB0E3EBC-6720-4E33-9219-95F4CEA6FA65

https://www.oracle.com/pls/topic/lookup?ctx=en/database/oracle/oracle-database/23&id=BRADV-GUID-54099115-7158-4ED4-A537-59451B3E14DC

L’article Oracle 23ai New Backup & Recovery Features est apparu en premier sur dbi Blog.

Introduction

The Oracle database Standard Edition do not have the Transparent Tablespace Encryption (TDE) feature which is a part of ASO (Advanced Security), and ASO is availabale only on Entreprise Edition version.

But encryption can still be used, with a little developpement effort.$

The test platform

I have an Oracle 19c SE2 version with one pdb (PDB11).

I will create a schema named safe, which will contains the encript/decrypt function and th key to be used to encrypt and decrypt.

Also the safe schema will contain a secret table with encrypted data.

On the database I will create an user scott, with minimal rights (only create session) which can read encrypted data.

Let’s build the environnement:

Connect on the pdb11 as sysdba and create the safe tablespace, and user safe.

SQL> alter session set container=pdb11;

Session altered.

SQL> create tablespace safe;

Tablespace created.

SQL> create user safe identified by tiger default tablespace safe;

User created.

SQL>  grant create session to safe;

Grant succeeded.

SQL> alter user safe quota unlimited on safe;

User altered.

SQL>  grant resource to safe;

Grant succeeded.

SQL> grant create view to safe;

Grant succeeded.

User safe need to have execution rights on dbms_crypto package

SQL>  grant execute on dbms_crypto to safe;

Grant succeeded.

From now on we are going to do all commands as safe user. Connect as user safe to the PDB11. and create the secret_keys table to hold all needed keys for decrypt.

[CDB01][oracle@db ~]$ sqlplus safe/tiger@db.sub05191447261.vcn01.oraclevcn.com:1521/pdb11

SQL> create table secret_keys(key_bytes_raw RAW (32), iv_raw RAW (16));

Table created.

Insert random keys in the secret_keys table.

Do not loose, or change these key. Your data will be lost.

SQL> insert into secret_keys(key_bytes_raw,iv_raw) values(DBMS_CRYPTO.RANDOMBYTES(256/8),DBMS_CRYPTO.RANDOMBYTES(16));

1 row created.

SQL> commit;

Commit complete.

SQL> select * from secret_keys;

KEY_BYTES_RAW
----------------------------------------------------------------
IV_RAW
--------------------------------
CCF9303B9AE706AC6212F9672FF8720113EC75D7C5FFAC267E42128AC40A33FA
A39F34AB9705CE91A6E6B4E592B32E28

Create the encrypt and decrypt functions.

SQL> CREATE OR REPLACE FUNCTION encr(input_string in VARCHAR2)
RETURN RAW
  IS 
    encryption_type    PLS_INTEGER := DBMS_CRYPTO.ENCRYPT_AES256
                       + DBMS_CRYPTO.CHAIN_CBC
                       + DBMS_CRYPTO.PAD_PKCS5;
    iv_raw             RAW (16);
    encrypted_raw      RAW (2000);
    key_bytes_raw      RAW (32);
    BEGIN
    --DBMS_OUTPUT.PUT_LINE ( 'Original string: ' || input_string);
    select key_bytes_raw,iv_raw into key_bytes_raw,iv_raw FROM secret_keys;
    encrypted_raw := DBMS_CRYPTO.ENCRYPT(
       src => UTL_I18N.STRING_TO_RAW (input_string,  'AL32UTF8'),
       typ => encryption_type,
       key => key_bytes_raw,
       iv  => iv_raw
    );
    --DBMS_OUTPUT.PUT_LINE ( 'Encrypted raw: ' || encrypted_raw);
    return encrypted_raw;
  END encr;
/

Function created.


SQL> CREATE OR REPLACE FUNCTION decr(encrypted_raw in RAW) 
return varchar2
IS 
  encryption_type    PLS_INTEGER := DBMS_CRYPTO.ENCRYPT_AES256
                 + DBMS_CRYPTO.CHAIN_CBC
                 + DBMS_CRYPTO.PAD_PKCS5;
  iv_raw             RAW (16);
  key_bytes_raw      RAW (32);
  decrypted_raw      RAW (2000);
BEGIN
  select key_bytes_raw,iv_raw into key_bytes_raw,iv_raw FROM secret_keys;
  decrypted_raw := DBMS_CRYPTO.DECRYPT(
    src => encrypted_raw,
    typ => encryption_type,
    key => key_bytes_raw,
    iv  => iv_raw
  );
  return UTL_I18N.RAW_TO_CHAR (decrypted_raw, 'AL32UTF8');
END decr;
/

Function created.

Test how these fonctions work:

-- encypt
SQL> select encr('Hello') from dual;

ENCR('HELLO')
--------------------------------------------------------------------------------
75388B4D940B559DD1A5D5BCA753101B

--decrypt
SQL> select decr('75388B4D940B559DD1A5D5BCA753101B') from dual;

DECR('75388B4D940B559DD1A5D5BCA753101B')
--------------------------------------------------------------------------------
Hello

--encrypt/decrypt
SQL> select decr(encr('Hello')) from dual;

DECR(ENCR('HELLO'))
--------------------------------------------------------------------------------
Hello

On the schema safe create the table that need to be protected, let’s call it, secret_emp, and insert an encrypted string.

SQL> create table secret_emp(name_encrypted RAW(2000));

Table created.

SQL> insert into secret_emp(name_encrypted) values(encr('Obama'));

1 row created.

SQL> commit;

Commit complete.

SQL> select * from secret_emp;

NAME_ENCRYPTED
--------------------------------------------------------------------------------
9115E6FD54D676DB043CD0D1BF982027

Finally we create a view on this table named authorized_read_emp. On this view authorised users will select to get the decrypted data.

SQL> create view authorized_read_emp as select decr(name_encrypted) name_decrypted from secret_emp;

View created.

SQL> select * from authorized_read_emp;

NAME_DECRYPTED
--------------------------------------------------------------------------------
Obama

Now we are going to create an user which has the rights to read the plan text data. For this user the encryption is transparent.

Connect as sysdba to the pdb11 and create an user scott. This user is allowed to read the data from authorized_read_emp view.

SQL> create user scott identified by tiger;

User created.

SQL> grant create session to scott;

Grant succeeded.

And finally as safe user, give the rights to user scott to read the view:

[CDB01][oracle@db ~]$  sqlplus safe/tiger@db.sub05191447261.vcn01.oraclevcn.com:1521/pdb11

SQL> grant select on authorized_read_emp to scott;

Grant succeeded.

SQL> exit; 

Back to user scott, let’s try to read the plan text data:

[CDB01][oracle@db ~]$  sqlplus scott/tigger@db.sub05191447261.vcn01.oraclevcn.com:1521/pdb11

SQL> select * from safe.authorized_read_emp;

NAME_DECRYPTED
--------------------------------------------------------------------------------
Obama

Conclusion

Ok is not really like the TDE implementation, and maybe a better implementation could be done. But the main points are:

• Schema ‘safe‘ contains only encrypted data. If someone stole the storage disks from the datacenter it cannot have access to tha data as it is stored encrypted. This is one of the main goal of TDE.

• Schema ‘safe‘ contains also the keys needed to decrypt the data. This is a limitation. The keys to decrypt data can be stored outside the database to increase the security.

• The user ‘scott‘ can access the data in a transparent mode as the data is decrypted on the fly. No data in plain is stored in the database.

• The user ‘scott‘ has only the rights to connect and to read the data which is decrypted on the fly.

L’article Simulate (Transparent Tablespace Encryption) TDE on Oracle Standard Edition est apparu en premier sur dbi Blog.

Introduction

WebHooks are HTTP callbacks that are triggered by events. In case of GitlLab these can be events like pushing some code, or creating an issue, or pushing some comments, create a merge request and so on.

Somehow, is a way for an application to send data, trigger events, in real time to another application, even if the target application don’t ask for the data.

In this post I will explain how to add a WebHook to a project and how to execute some external code when en event is triggered.

I will use an Flask application, which will be triggered by GitLab WebHook.

VM configuration

The test VM is created on the OCI cloud. It is a RedHat VM but that is not very important.

First I have to open the port 8200 (used by Flask). RedHat use by default firewalld as firewall:

# add 8200 port using tcp to the public zone
[root@wb ~]# firewall-cmd --zone=public --add-port=8200/tcp
success
# list opened ports
[root@wb ~]# firewall-cmd --list-all
public (active)
target: default
icmp-block-inversion: no
interfaces: ens3
sources:
services: dhcpv6-client http ssh
ports: 8200/tcp
protocols:
forward: no
masquerade: no
forward-ports:
source-ports:
icmp-blocks:
rich rules:

The Flask application

For example I create a small Flask application which is waiting for an event on 8200 port. The easiest application is to retrieve the HTTP frame.

# app.py 
from flask import Flask, request, abort
import json

app = Flask(__name__)

@app.route('/my_webhook', methods=['POST'])
def my_webhook():
    if request.method == 'POST':
        print(json.dumps(request.json, indent=2))
        return 'success', 200
    else:
        abort(400)

if __name__ == '__main__':
    app.run(host='0.0.0.0', port=8200, debug=True)

The Flask will launch a http server which is listen on all interfaces (0.0.0.0) en on port 8200
The path to be used in the HTTP POST call is my_webhook
Also I used the debug mode to get the most traces.
The Flask application is waiting a POST request (line 10). And answer success.
Otherwise, return 400 HTTP code.

Start the Flask application

(venv) [opc@wb ~]$ python app.py
 * Serving Flask app 'app' (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: on
 * Running on all addresses.
   WARNING: This is a development server. Do not use it in a production deployment.
 * Running on http://172.168.0.40:8200/ (Press CTRL+C to quit)
 * Restarting with stat
 * Debugger is active!
 * Debugger PIN: 295-390-357

GitLab configuration

You must have the role maitainer on your project, to have the rights to manage WebHooks.

Select Settings -> WebHooks from the left Menu

Once The WebHook was saved it can be tested. From Test button you can choose an event type to test.

Let’s try with a push event:

When the event is triggered in GitLab, the Flask application must dump the http frame on the screen:

 * Running on http://172.168.0.40:8200/ (Press CTRL+C to quit)
 * Restarting with stat
 * Debugger is active!
 * Debugger PIN: 295-390-357
.....
{
  "object_kind": "push",   
  "event_name": "push",
  "before": "6aa30335b09*****53183fb28bb84aada5cf",
  "after": "a48e187ef217a******62498378e5023d355",
  "ref": "refs/heads/main",
  "checkout_sha": "a48e187ef21********9490562498378e5023d355",
  "message": null,
  "user_id": 11774125,
  "user_name": "your username ",
  "user_username": "your.usernam",
  "user_email": "",
  "user_avatar": "https://gitlab.com/uploads/-/system/user/avatar/11774125/avatar.png",
  "project_id": 38878393,
  "project": {
    "id": 38878393,
    .....

On line 7 we can see the push event from GitLab.

Conclusion

The use of WebHooks from GitLab to run external code represents another facility in DevOps tools.

Their implementation is easy and immediate.

An example of use case is the integration with Ansible Automation Controller, to run Ansible jobs.

L’article Using WebHooks with GitLab to launch external tools est apparu en premier sur dbi Blog.

Introduction

When Oracle database is configured with Oracle Key Vault, all mater encryption key (MEK) are stored on Oracle Key Vault server.

Rekey is the operation of changing the MEK.

In the previous article Clone Oracle Database configured with Oracle Key Vault (OKV) I cloned a database CDB01 to CDB02 configured with OKV. At the end of the clone process the cloned database CDB02 use the same keys as the source database. In a production environment this is not an acceptable solution. The cloned CDB02 database (which can be a clone for test purpose), need to use it’s own keys. To achieve this goal we need to REKEY the CDB02 database.

First we are going to create a wallet for CDB02.

The we are going execute the REKEY operation, to generate new master encryption keys.
At the end to make the full separation between CDB01 and CDB02 we remove the rights for CDB02 to read the wallet of CDB01.

Preparation

As explained in the previous post, the RESTFul api is installed in /home/oracle/okv

I use a script to set the RESTFul API environnement:

[oracle@db okv]$ cat /home/oracle/okv/set_okv_rest_env.sh
export OKV_RESTCLI_CONFIG=$HOME/okv/conf
export JAVA_HOME=/usr/java/jdk-11.0.10
export OKV_HOME=$HOME/okv
export PATH=$PATH:$OKV_HOME/bin

[oracle@db okv]$ source /home/oracle/okv/set_okv_rest_env.sh

I use an SQL script to output the wallet status:

[oracle@db okv]$ cat $HOME/tde.sql
set pages 200
set line 300
col WRL_PARAMETER format a50
col status forma a10
col pdb_name  format a20
select pdb_id, pdb_name, guid from dba_pdbs;
select * from v$encryption_wallet where con_id != 2;

The initial status of CDB02

[oracle@db ~]$ . oraenv <<< CDB02
[CDB02][oracle@db ~]$ sqlplus / as sysdba

SQL> show parameter wallet_root

NAME				     TYPE	 VALUE
------------------------------------ ----------- ------------------------------
wallet_root			     string	 /opt/oracle/admin/CDB02/wallet

SQL> show parameter tde_configuration

NAME				     TYPE	 VALUE
------------------------------------ ----------- ------------------------------
tde_configuration		     string	 KEYSTORE_CONFIGURATION=OKV|FIL

SQL> @tde.sql

    PDB_ID PDB_NAME		GUID
---------- -------------------- --------------------------------
	 3 PDB01		0AE3AEC4EE5ACDB1E063A001A8ACB8BB
	 2 PDB$SEED		0AE38C7FF01EC651E063A001A8AC821E


WRL_TYPE  WRL_PARAMETER                       STATUS              WALLET_TYPE   WALLET_OR KEYSTORE FULLY_BAC CON_ID
-------- ----------------------------------- -------------------- ------------- --------- -------- --------- -------
FILE     /opt/oracle/admin/CDB02/wallet/tde/ OPEN_NO_MASTER_KEY    AUTOLOGIN    SINGLE      NONE     UNDEFINED  1
OKV                                          OPEN                  OKV          SINGLE      NONE     UNDEFINED  1
FILE                                         OPEN_NO_MASTER_KEY    AUTOLOGIN    SINGLE      UNITED   UNDEFINED  3
OKV                                          OPEN_UNKNONW_         OKV          SINGLE      UNITED   UNDEFINED  3

SQL> exit; 

[CDB02][oracle@db ~]$ /opt/oracle/admin/CDB02/wallet/okv/bin/okvutil list
Enter Oracle Key Vault endpoint password: endpoint_password
Unique ID                               Type            Identifier
600D0743-01D9-4F2F-BF6F-C9E8AC74FF2A	Symmetric Key	TDE Master Encryption Key: TAG CDB:CDB01 MEK first
6A752388-F93D-4F14-BF35-39E674CAAFED	Symmetric Key	TDE Master Encryption Key: TAG REKEY CDB01
AB294686-1FC4-4FE8-BFAD-F56BAD0A124B	Symmetric Key	TDE Master Encryption Key: TAG REKEY CDB01
BB0CC77A-10AD-4F55-BF0A-9F5A4C7F98C1	Symmetric Key	TDE Master Encryption Key: TAG CDB:DBTDEOKV:PDB1 MEK first

Create a wallet for CDB02

[CDB02][oracle@db json]$ $OKV_HOME/bin/okv manage-access wallet create --generate-json-input > create_db_wallet_CDB02.json
[CDB02][oracle@db json]$ cat create_db_wallet_CDB02.json
{
  "service" : {
    "category" : "manage-access",
    "resource" : "wallet",
    "action" : "create",
    "options" : {
      "wallet" : "ORA_CLONES",
      "type" : "GENERAL",
      "description" : "Wallet for Oracle Clones"
    }
  }
}

[CDB02][oracle@db json]$ $OKV_HOME/bin/okv manage-access wallet create --from-json create_db_wallet_CDB02.json
{
  "result" : "Success"
}

Set the default wallet for CDB02

[CDB02][oracle@db json]$ okv manage-access wallet set-default --generate-json-input > set_default_wallet_CDB02.json


[CDB02][oracle@db json]$ cat set_default_wallet_CDB02.json
{
  "service" : {
    "category" : "manage-access",
    "resource" : "wallet",
    "action" : "set-default",
    "options" : {
      "wallet" : "ORA_CLONES",
      "endpoint" : "DB_CDB02",
      "unique" : "FALSE"
    }
  }
}

[CDB02][oracle@db json]$ okv manage-access wallet set-default --from-json set_default_wallet_CDB02.json
{
  "result" : "Success"
}

# test 
[CDB02][oracle@db json]$ $OKV_HOME/bin/okv manage-access wallet get-default --endpoint DB_CDB02
{
  "result" : "Success",
  "value" : {
    "defaultWallet" : "ORA_CLONES"
  }
}

# list wallets access for endpoint DB_CDB02
[CDB02][oracle@db json]$ $OKV_HOME/bin/okv manage-access wallet list-endpoint-wallets --endpoint DB_CDB02
{
  "result" : "Success",
  "value" : {
    "wallets" : [ "ORA_CLONES", "ORA_DB" ]
  }
}

REKEY operation

[CDB02][oracle@db json]$ sqlplus / as sysdba 

-- list all keys for CDB02
SQL> set line 200
SQL> col key_id format a40;
SQL> select KEY_ID, KEYSTORE_TYPE,CREATION_TIME from V$ENCRYPTION_KEYS;

KEY_ID                                   KEYSTORE_TYPE	   CREATION_TIME
---------------------------------------- ----------------- ----------------------------
066477563C41354F9ABFFD71C439728D90	 OKV		   12-MAR-24 11.38.29.789446 AM +00:00
06389A1CCF31E64F17BFC1101D9700F83E	 OKV		   12-MAR-24 11.53.46.361951 AM +00:00
064A92E70C7DBB4FBCBFDE46A9226CFB0A	 OKV		   12-MAR-24 11.53.45.932774 AM +00:00
06FED2B8DA29444F57BF11BB545ED7E60D	 OKV		   12-MAR-24 11.20.59.949238 AM +00:00 

SQL> ADMINISTER KEY MANAGEMENT SET ENCRYPTION KEY FORCE KEYSTORE IDENTIFIED BY "endpoint_password" container=all;

Remove access from CDB01 wallet:

[CDB02][oracle@db json]$ $OKV_HOME/bin/okv manage-access wallet remove-access --generate-json-input > remove_access_walet_CDB02.json

[CDB02][oracle@db json]$ cat remove_access_walet_CDB02.json
{
  "service" : {
    "category" : "manage-access",
    "resource" : "wallet",
    "action" : "remove-access",
    "options" : {
      "wallet" : "ORA_DB",
      "endpoint" : "DB_CDB02"
    }
  }
}

[CDB02][oracle@db json]$ $OKV_HOME/bin/okv manage-access wallet remove-access --from-json remove_access_walet_CDB02.json
{
  "result" : "Success"
}

[CDB02][oracle@db json]$ /opt/oracle/admin/CDB02/wallet/okv/bin/okvutil list
Enter Oracle Key Vault endpoint password:
Unique ID                               Type            Identifier
1B382343-A786-4F26-BFF9-35A8329A327C	Symmetric Key	TDE Master Encryption Key: MKID 0612F89A18C7984F27BF571A0420C58025
52B62409-6E8D-4F6F-BF08-F7DD73EC1938	Symmetric Key	TDE Master Encryption Key: MKID 06A9FD621A85A74F46BFD88BEB6082B9EB
2DE4025E-CF35-454D-9F60-33640DAAC067	Template	Default template for DB_CDB02

-- restart CDB02 to test if the database open withouth any issue
SQL> startup force

SQL> @$HOME/tde.sql

    PDB_ID PDB_NAME		GUID
---------- -------------------- --------------------------------
	 3 PDB01		0AE3AEC4EE5ACDB1E063A001A8ACB8BB
	 2 PDB$SEED		0AE38C7FF01EC651E063A001A8AC821E


WRL_TYPE    WRL_PARAMETER                       STATUS             WALLET_TYPE WALLET_OR KEYSTORE FULLY_BAC     CON_ID
----------- ----------------------------------- ------------------ ----------- --------- -------- --------- ----------
FILE        /opt/oracle/admin/CDB02/wallet/tde/ OPEN_NO_MASTER_KEY AUTOLOGIN   SINGLE      NONE       UNDEFINED          1
OKV                                             OPEN               OKV         SINGLE      NONE       UNDEFINED          1
FILE                                            OPEN_NO_MASTER_KEY AUTOLOGIN   SINGLE      UNITED     UNDEFINED          3
OKV                                             OPEN               OKV         SINGLE      NONE       UNDEFINED          3

Database CDB02 open correctly.

List the accessible keys for CDB02:

[CDB02][oracle@db json]$ /opt/oracle/admin/CDB02/wallet/okv/bin/okvutil list
Enter Oracle Key Vault endpoint password:
Unique ID                               Type            Identifier
1B382343-A786-4F26-BFF9-35A8329A327C	Symmetric Key	TDE Master Encryption Key: MKID 0612F89A18C7984F27BF571A0420C58025
2DE4025E-CF35-454D-9F60-33640DAAC067	Template	Default template for DB_CDB02

List the wallets accessible for CDB02:

[CDB02][oracle@db json]$ $OKV_HOME/bin/okv manage-access wallet list-endpoint-wallets --endpoint DB_CDB02
{
  "result" : "Success",
  "value" : {
    "wallets" : [ "ORA_CLONES" ]
  }
}

CDB02 has no more access to the CD01 wallet.

Conclusion

If the database is not configured with OKV, after a REKEY operation the wallet file, stored on local disk, must be saved. When the database is configured with OKV when a REKEY operation is issued we have to do …. nothing. The keys are automatically stored in OKV database without any intervention. Just only one remark. The endpoint, in our example DB_CDB02, need to have a default wallet configured. Otherwise the keys will not belongs to any wallet. That doesn’t mean that the CDB02 database cannot access them, but having keys outside wallets in OKV, increase the maintenance operations.

L’article REKEY operation on Oracle Database configured with Oracle Key Vault est apparu en premier sur dbi Blog.