YaK core Multi-Platform open source Automation Tool simplifies the deployment of Ansible playbooks through a clean UI and API. It offers an intuitive interface where users can upload playbooks, configure parameters, and deploy them seamlessly across various platforms, and all managed through a centralized inventory stored in a PostgreSQL database. With YaK Core, developers can focus on writing application code without worrying about infrastructure setup or management.

YaK consists of two parts: YaK Core, which is open source, and YaK Components, which can be installed on top. These YaK Components are platform-agnostic service packages (e.g., PostgreSQL, Oracle DB, MongoDB, Kubernetes, etc.), written in Ansible by experts. They provide essential operational features such as backup, patching, upgrades, and high availability. If you’d like to learn more about the available YaK components, feel free to contact us!

But that’s not all. YaK Core also lets you create your own YaK Components . Once created, your component becomes immediately available for deployment across all platforms supported by YaK Core.

In this blog, I’ll show you how easy it is to create your own YaK Component using Ansible, upload it to YaK Core, and deploy it across any supported platform.

YaK Demo platform provisioning

To get started with YaK Core Multi-Platform open source solution, visit https://yak4all.io and provision your own YaK demo environment (take 5 minutes to be ready).

Build your YaK Component

To build a YaK Component, you need to declare at least the following three files

1. playbooks/create_linux_users.yml
2. manifest.yml
3. yak_variables_specifications/basic_variables_specifications.yml

1. The Ansible Playbook

playbooks/create_linux_users.yml
This file is simply your Ansible playbook, nothing more. The only requirement is that the code uses variables, which will be exposed in the UI for configuration. The example playbook below will create a user and optionally grant them sudo privileges.

---
- name: Create Linux users
  hosts: linux_hosts
  become: true
  gather_facts: true

  tasks:
    - debug:
        var: user

    - name: Create users
      ansible.builtin.user:
        name: "{{ item.username }}"
        create_home: "{{ item.create_home | default(true) }}"
        state: present
      loop: "{{ user }}"

    - name: Add users to sudoers
      community.general.sudoers:
        name: "yak-sudoer-{{ item.username }}"
        user: "{{ item.username }}"
        commands: ALL
        state: present
      loop: "{{ user }}"
      when: item.is_sudoer
        
  post_tasks:
    - name: Update component state
      delegate_to: localhost
      yak.core.yak_component_state_update:
        component_state_name: 'deployed'
...

2. Manifest file

manifest.yml
This file contains the basic information about your component and specifies which playbooks can be executed.

name: linux_users

version:
  major: 1
  minor: 0
  patch: 0

sub_component_types:
  - display_label: Linux users
    name: create_linux_users
    features:
      - display_label: Create Linux users
        name: create_linux_users
        playbook_name: playbooks/create_linux_users.yml

    inventory_maps:
      - group_name: linux_hosts
        group_nicename: Linux hosts
        group_description: Host on which the users will be created
        group_min_hosts: 1
        group_max_hosts: 100
        type: host
        os_type: Linux

3. Variable specification file

yak_variables_specifications/basic_variables_specifications.yml
Now you can define and provide all the specifications for the variables you want to make configurable, with all the required settings .

- variableName: user
  niceName: Users to create
  dataType: array
  children:
    - variableName: username
      niceName: Username
      dataType: string
      mandatory: true
      defaultValue: yak
      isOneOffSetting: false
      usage: Name of the user to create

    - variableName: create_home
      niceName: Create Home directory
      dataType: boolean
      mandatory: true
      defaultValue: true
      isOneOffSetting: false
      usage: Tick the box if you want to create a Home directory for the user (/home/<username>)

    - variableName: is_sudoer
      niceName: Grant sudo privileges
      dataType: boolean
      mandatory: true
      defaultValue: true
      isOneOffSetting: false
      usage: Tick the box if you want to grant "ALL" privileges escalation to the user

That’s it! You now have all the necessary files for your first YaK Component. Next, create a ZIP package and upload it to your deployed YaK Demo environment.

To make things easier, I’ve created a ZIP file that you can upload directly. : create_linux_user.zip

Setup a Server

For this task, simply follow the documentation below up to Step 4: Deploy your server https://dbi-services.gitbook.io/yak-user-doc/introduction/yak-demo

Declare and deploy your Component

You’re now ready to declare and deploy your component!

1. Declare

YaK UI -> Components -> Declare -> Component_type : linux_users -> Save

2 Deploy

YaK UI -> Components -> Select LinuxUser -> Action -> Create Linux User -> Confirm

Conclusion

This component can now be deployed on any cloud platform or integrated on On-Premises environment using the YaK UI, and can also be deployed in parallel on up to 100 servers, as specified in your Manifest file.

With this solution, you can provide your colleagues with an intuitive and efficient way to work with Ansible playbooks, enhancing their overall experience .

For more Information about YaK see the blogs available here : https://www.dbi-services.com/blog/yak

L’article YaK Core – The Holy Grail for Deploying Ansible Code Everywhere est apparu en premier sur dbi Blog.

Delphix Data Virtualization Creates only a virtual copy of your database blocks instead of a full physical copy. For example this allows making a new big database available only in a few seconds.

Delphix Data Masking Protects sensitive data by replacing it with realistic but non-sensitive values, ensuring compliance with privacy regulations while maintaining data usability for development, testing, and analytics. For reference, dbi-services has developed Swiss masking rules for customers to apply all their specific needs.

Setup

As always, setting up the prerequisites took some time, but once the environment was built, it was truly rewarding to demonstrate the combined power of these tools to my colleagues. It’s a real pleasure, like a game, building database environments in a flash, where the data is directly masked.

This allows you to rapidly provision databases to your internal or external developers, without taking any risk of a data phishing attack!

Watch this video! It’s only a minute-long and you’ll see how easy it is! to create a new masked Oracle pluggable database VPDB2_masked on another server (These tasks can also be fully automated in a pipeline using an API)

Relevant outcomes

The Delphix pricing model is based on the size of your databases, thus you can apply the same Virtualization and Masking capabilities at least for PostgreSQL, SQL Server, and Sybase at no additional cost. Are you interested in making your database development environments fully secure? Don’t wait, dbi services has your solution!

For those who are interested in more details

1. Table data from the source database PDB1

2. Table data in the new created and masked virtual database VPDB2_Masked

3. Output of the masking job

Useful information related to Delphix

Blog that presents Data Masking in detail

Customer success story

Webinar that presents Data Virtualization

L’article Delphix Data Virtualization and Masking est apparu en premier sur dbi Blog.

Tuesday evening we started with the traditional “Schweizer Abend” Dinner with the Swiss community. We had a special guest at our table with Frits Hoogland who comes from Holland/Netherlands. Fritz also made the switch from Oracle to PostgreSQL a few years ago, and now works for Yugabyte. The dinner was also an opportunity for us to discuss current IT trends.

Wednesday Morning the conference started and it was amazing to see how many people were there. Below is a picture of the plenary session at the start of the conference.

dbi services was present at the conference with 1 workshop and 2 presentations:

Daniel Westermann – 1 Day Workshop PostgreSQL DBA Essentials

Julia Gugel – Hero + Hero = SuperHero !

Daniel Westermann – TOAST, for breakfast and compression

As you can see, Daniel and Julia’s sessions were quite busy.

YaK

This event was also an opportunity for dbi services to present our new open source product that we just launched a month ago: the YaK. The discussions around this product during the 3 days were very interesting and I could gather a lot of new development ideas/proposals to enhance the product.

I also had the opportunity to present the product in the event’s Sponsors Track.

Hervé Schweitzer – YaK – The Next-Gen Multi-Platform PostgreSQL Database Managed Services

Last not least – dbi services’ booth

I would also like to thank Martin Moll, Region Manager Zürich at dbi services, who successfully managed the booth during the 3 days, which is not an easy task.

See you there next year for many new interesting sessions.

L’article 2022.pgconf.eu Berlin conference for tech est apparu en premier sur dbi Blog.

It all began with a challenge

In the consulting job, some tasks can be a bit repetitive. They also take up valuable consulting time that the client would be more than happy to use for higher value-added tasks. So, my team and I started thinking about a solution two years ago. Which tasks could be automated to facilitate the work of our consultants and speed up the delivery to the client? The deployment of infrastructures. Of course!

Then COVID came along. It was a disaster for everyone… but this pandemic gave us the opportunity and the time to start the discussion: How could we implement a IaC (Infrastructure as Code) solution usable for all on-premises platforms and for public Cloud providers. The idea behind this question? Deploy multi-technology infrastructures in a minimum of time and effort including all our Best Practices and satisfy our consultants and customers. A great way to kill two birds with one stone.

The bumpy road from the idea to the realization

The idea is there, the feedbacks are positive, everyone is enthusiastic. Next step: the development. This is where the first big questions came in: Should we use the services provided by the Cloud providers to deploy infrastructures with IaC? Nope, because that meant managing different solutions for each Cloud provider. Not efficient. Another corollary: you would have needed as many experts as solutions. At a time of IT labor shortage? Not efficient either.

But we are not afraid of a challenge. We pushed the thought further and found a solution: using Ansible for the deployment! Why Ansible? Because it fully met all our requirements and, icing on the cake, it also has many collections/roles for the different Cloud providers. You gotta Ansible. The idea was good on paper. But the practice quickly turned out to be “bumpier” than expected. One of the problems we encountered was that the different inventories could not talk to each other. So, we had to switch from one inventory to another, and could not get an overview of all the components deployed at the different Cloud providers. Additionally, each inventory had its own structure making the code too complex to manage.

The YaK is born!

The taste of challenge runs in our veins. So, we put our heads together and decided that you’re never better served than by yourself. Result: we developed our own product with our own Ansible inventory. Our IaC solution was born!

The happy moms and dads were so moved that they had a tough time finding a name to match their happiness. At dbi services, we love IT and simple things: We have developed an IaC solution, why not simply call it YaK? It suits us all the more since the YaK is a solid, reliable… and adorable animal, isn’t it? All that was left was to convince our marketing team to follow our lead and come up with a logo.

We fully trust the potential of our YaK. We now have to convince you too. So, we have opened its core part to the Open Source community. It is all yours to test. See you here: https://www.dbi-services.com/products/yak/

I also take this opportunity to thank Jérôme Witt, Nicolas Penot, David Hueber, Joël Cattin and Ali Ashgari who have been part of the project since the beginning. Many others joined us since: Nicolas Jardot, Nicolas Meunier, Chay Te, Jean-Philippe Clapot, Pascal Brand, Olivier Spiesser, Alexandre Nestor, Alain Fuhrer… and it’s not finished #besharing #besuccessful #bdbi

L’article How the YaK is born est apparu en premier sur dbi Blog.

First test : table creation

We create 2 tables, one with the default Postgres storage engine HEAP, and one with the new storage enfine ZHEAP.

PSQL> create table heap2 as select a.*, md5(a::varchar), now() from generate_series(1,5000000) a;
 
SELECT 5000000
Time: 12819.369 ms (00:12.819)

PSQL> create table zheap2  with (storage_engine='zheap') as select a.*, md5(a::varchar), now() from generate_series(1,5000000) a;
SELECT 5000000
Time: 19155.004 ms (00:19.155)

You noticed, that with Postgres you can choose your storage engine at table level :-). The table creation with ZHEAP is slower, but is is normal because now we have to create the UNDO segment also.

Second test : Size of the both tables

Before to start the tests we will check the size of the HEAP and ZHEAP tables, as announced yesterday the HEAP table should be smaller, because we have less header information.

PSQL>  select pg_size_pretty(pg_relation_size('heap2'));
 pg_size_pretty 
----------------
 365 MB
PSQL> select pg_size_pretty(pg_relation_size('zheap2'));
 pg_size_pretty 
----------------
 289 MB

The ZHEAP tables is smaller, it exactly what Amit explain us yesterday, because the block header with ZHEAP is smaller. If you want to learn more read his presentation from yesterday. Again the link is on my blog from yesterday.

Third test : Update on the table

To get the bloat effect on the HEAP table, we will now update the full table and see what happen.

PSQL> update heap2 set a=a+12222222;
UPDATE 5000000
Time: 19834.911 ms (00:19.835)

PSQL> update zheap2 set a=a+12222222;
UPDATE 5000000
Time: 26956.043 ms (00:26.956)

PSQL> select pg_size_pretty(pg_relation_size('zheap2'));
 pg_size_pretty 
----------------
 289 MB
PSQL> vacuum heap2;
PSQL> select pg_size_pretty(pg_relation_size('heap2'));
 pg_size_pretty 
----------------
 730 MB

The same as for the creation the update is a bit longer, but the update with ZHEAP write many information into the log file.We should test again this update with disabling the writing of information into the log file about creating undo segment.
But as you can see, the most important information here is that the table don’t bloat as the HEAP table, now the HEAP table is 2 times bigger despite I executed a VACUUM.

Fourth test: test of the ROLLBACK

To test the ROLLBACK we have to open first a transaction with BEGIN;

PSQL>  begin;
BEGIN
PSQL>* update heap2 set a=a+12222222;
UPDATE 5000000
Time: 22071.462 ms (00:22.071)
PSQL> * rollback;
ROLLBACK
Time: 1.437 ms

PSQL> begin;
BEGIN
PSQL> * update zheap2 set a=a+12222222;
UPDATE 5000000
Time: 28210.845 ms (00:28.211)
PSQL> * rollback;
ROLLBACK
Time: 0.567 ms

This is the part where I’m the most surprised, the ROLLBACK for ZHEAP is so fast as for HEAP, I can’t explain that. I will leave my colleague Daniel Westermann making deeper tests :-). Because with ZHEAP he has to apply the undo blocks, where HEAP tables only mark the transactions as aborted.

Fifth tests : Check of the query performances

For this test we have to first flush the filesystem cache and to restart the database, to be sure that nothing is cached.

postgres@dbi-pg-tun:/home/postgres/zheap/ [ZHEAP] pgstop 
waiting for server to shut down.... done
server stopped

postgres@dbi-pg-tun:/home/postgres/ [ZHEAP] sudo sync
postgres@dbi-pg-tun:/home/postgres/ [ZHEAP] sudo echo 3 > /proc/sys/vm/drop_caches

postgres@dbi-pg-tun:/home/postgres/zheap/ [ZHEAP] pgstart
waiting for server to start.... done
server started

Now we are ready for the last test

postgres@dbi-pg-tun:/home/postgres/zheap/ [ZHEAP] sqh
PSQL> select count(*) from heap2;
  count  
---------
 5000000
Time: 3444.869 ms (00:03.445)

PSQL> select count(*) from zheap2;
  count  
---------
 5000000
Time: 593.894 ms

As you can see the query performance are improved significantly for full table scan :-), because the table didn’t bloat as for the HEAP table. For you information I started additionally 2 times a full update before to restart the database and the HEAP table is now 3 times bigger.

PSQL> select pg_size_pretty(pg_relation_size('heap2'));
 pg_size_pretty 
----------------
 1095 MB

Time: 0.508 ms
PSQL> select pg_size_pretty(pg_relation_size('zheap2'));
 pg_size_pretty 
----------------
 289 MB

Conclusion of these tests

• Postgres allow the usage or not of UNDO’s at the table level
• We are surprise how fast the ROLLBACK are, but this must be tested again, I don’t understand why
• Select performance are improved significantly for full table scan
• The storage will not bloat anymore with ZHEAP
• Finally only the updates are a little bit slower

It will be interesting to follow the discussions around this feature on the mailing list.

L’article Deep dive Postgres at the #pgconfeu conference est apparu en premier sur dbi Blog.

The #SwissPGday is located in Rapperswil at the HSR(Hochschule für Technik) near the Zürisee, it’s a nice location and everything was nicely organized. Also the look-out from the HSR is very beautiful.

After a short introduction from Stephan Keller (HSR) where the room of the plenum session was completely full, the presentations was splitted in two streams. For your information last year they had around 60 participants this year we are around 100 participants, so I would say, that PostgreSQL it’s the trend.

Today what I especially appreciated, is that many teacher’s of the event are also developers of the PostgreSQL community, thus we get the latest information of some development and it was possible to discuss about new features directly with them.

I will also give you feedback of 2 sessions from other partner which I really appreciated. For sure the session from my colleague Daniel Westermann was the best one :-), who presented the new features of PostgreSQL 11 with many demos.

The first one was from Laurenz Albe(Cybertec), who presented a community Tool ORA_MIGRATOR and tips to migrate from Oracle to PostgreSQL.
The second one was from  Harald Armin Massa(2ndQuadrant), who presented the new PostgreSQL logical replication, which was developed by his company and is now partially available for the community edition.

As usual to finish an Apero was organized to drink a beer together and exchange with the other participants.

See you next year at the #SwissPGDay 2019.

L’article #SwissPGDay is a place where you can meet developers of the PostgreSQL community est apparu en premier sur dbi Blog.

This week I worked on a GoldenGate 12.1.2.1.10 POC setup and was facing an issue which for me is a serious drawback of the Oracle GoldenGate product.If you want to create additional columns on the target database online in a GoldenGate configuration you have to be aware of the below situation which can happen in your setup:The below demo was created on a Oracle GoldenGate Downstream server

For the test, I have created the schema scott/tiger on both the source and target databases, thus no initial load is needed

Source>@utlsampl.sql
Target>@utlsampl.sql

GGSCI (srv01) 1> dblogin useridalias ggsource
Successfully logged into database.

GGSCI (srv01 as goldengate@DB1) 2> miningdblogin useridalias ggcap
Successfully logged into mining database.

GGSCI (srv01 as goldengate@DB1) 3> register extract scott database
Extract SCOTT successfully registered with database at SCN 277324431694.

GGSCI (srv01 as goldengate@DB1) 5> add extract scott integrated tranlog, begin now
EXTRACT added.

GGSCI (srv01 as goldengate@DB1) 6> add trandata scott.emp

GGSCI (srv01 as goldengate@DB1) 6> info all

Program     Status      Group       Lag at Chkpt  Time Since Chkpt

MANAGER     RUNNING
EXTRACT     STOPPED     SCOTT       00:00:00      00:00:07

GGSCI (srv01 as goldengate@DB1) 7> add exttrail /u01/directories/ggtrail/POCGGP15/es, extract SCOTT
EXTTRAIL added.

GGSCI (srv01 as goldengate@DB1) 12> view params scott

EXTRACT SCOTT
USERIDALIAS ggsource
DBOPTIONS ALLOWUNUSEDCOLUMN
DDL INCLUDE ALL
TRANLOGOPTIONS MININGUSERALIAS ggcap
TRANLOGOPTIONS INTEGRATEDPARAMS (downstream_real_time_mine Y)
EXTTRAIL /u01/directories/ggtrail/POCGGP15/es
TABLE SCOTT.EMP;

GGSCI (srv01 as goldengate@DB1) 4> start extract scott

Sending START request to MANAGER ...
EXTRACT SCOTT starting

GGSCI (srv01) 1> add replicat repscott, exttrail /u01/directories/ggtrail/POCGGP15/es
REPLICAT added.

GGSCI (srv01 as goldengate@DB2) 10> view params repscott

REPLICAT REPSCOTT
useridalias ggtarget
DISCARDFILE /u01/app/goldengate/product/12.1.2.1/discard/REPSCOTT_DISCARD.txt,APPEND,megabytes 10
ASSUMETARGETDEFS
DBOPTIONS NOSUPPRESSTRIGGERS
MAP SCOTT.EMP,TARGET SCOTT.EMP;

GGSCI (srv01) 3> dblogin useridalias ggtarget
Successfully logged into database.

Extract current_scn on source database

sys@GMAS2> select current_scn from v$database;

   CURRENT_SCN
--------------
  277324550446

GGSCI (srv01 as goldengate@DB2) 5> start replicat repscott, afterscn 277324550446

Sending START request to MANAGER ...
REPLICAT REPSCOTT starting

GGSCI (srv01 as goldengate@DB2) 9> info all

Program     Status      Group       Lag at Chkpt  Time Since Chkpt

MANAGER     RUNNING
EXTRACT     RUNNING     SCOTT           00:00:03      00:00:05
REPLICAT    RUNNING     REPSCOTT        00:00:00      00:02:08

scott@DB2> alter table emp add TARGET_COL varchar(10) default null;

Table altered.

scott@DB1> alter table emp add SOURCE_COL varchar(10) default null;

Table altered.

scott@DB2> select ename,target_col,source_col from emp;

ENAME      TARGET_COL SOURCE_COL
---------- ---------- ----------
SMITH
ALLEN
WARD
...

scott@DB1> select ename, source_col from emp;

ENAME      SOURCE_COL
---------- ----------
SMITH
ALLEN
WARD
...

scott@DB1> update emp set source_col='change';

14 rows updated.

scott@DB1> commit;

Commit complete.

scott@DB1> select ename, source_col from emp;

ENAME      SOURCE_COL
---------- ----------
SMITH      change
ALLEN      change
WARD       change

scott@DB2> select ename,target_col,source_col from emp;

ENAME      TARGET_COL SOURCE_COL
---------- ---------- ----------
SMITH      change
ALLEN      change
WARD       change

Conclusion: A Solution exist: You’ll have to install Oracle GoldenGate release 12.2 which is available since this week, next blog with the solution will come soon

L’article With Oracle Goldengate take care of additional column creation on the replicated database est apparu en premier sur dbi Blog.

In the example below, tables from schema G001 will be added to an existing replication setup from the database PROD1 to REP1.

CFG_ADV_COND
CFG_NARRATIVE_TEMPLATE
CFG_REG_REPORT_RULES
CMN_LOOKUP
CMN_USER_LOGIN

Stop the replication environment

• Connect to the source database PROD1 and stop all extract groups for the replication to the database REP1

oracle@server1:~/ [PROD1] PROD1
oracle@server1:~/ [PROD1] cdgh
oracle@server1:/u99/app/goldengate/gss/11.1.1.1.0/ [PROD1] ggsci

GGSCI (server1) 1> info all

Program Status Group Lag Time Since Chkpt
MANAGER RUNNING
EXTRACT RUNNING DPG001 00:00:00 00:00:06
EXTRACT RUNNING G001 00:00:00 00:00:05

GGSCI (server1) 2> stop extract *
Program Status Group Lag Time Since Chkpt

MANAGER RUNNING
EXTRACT STOPPED DPG001 00:00:00 00:00:01
EXTRACT STOPPED G001 00:00:00 00:00:13

• Connect to the target database REP1 and stop the replicat group coming from the database PROD1

GGSCI (server2) 1> info all

Program Status Group Lag Time Since Chkpt
MANAGER RUNNING
REPLICAT RUNNING PROD1 00:00:00 00:00:04

GGSCI (server2) 2> stop REPLICAT PROD1
GGSCI (server2) 3> info all

Program Status Group Lag Time Since Chkpt
MANAGER RUNNING
REPLICAT STOPPED PROD1 00:00:00 00:00:04

Update the extract environment with the new tables

• Now we can update the extract group G001 on the source database with the new tables to be replicated. Edit the G001 parameter file and add the new tables (vi editor)

GGSCI (server1) 30> edit params G001
. . . 
table G001SCHEMA.CFG_ADV_COND;
table G001SCHEMA.CFG_NARRATIVE_TEMPLATE;
table G001SCHEMA.CFG_REG_REPORT_RULES;
table G001SCHEMA.CMN_LOOKUP;
table G001SCHEMA.CMN_USER_LOGIN;

• Add supplemental login on source database PROD1 for the new added tables

GGSCI (server1) 7> DBLOGIN userid goldengate, password ******
Successfully logged into database.

GGSCI (server1) 8> add trandata G001SCHEMA.CFG_ADV_COND
GGSCI (server1) 9> add trandata G001SCHEMA.CFG_NARRATIVE_TEMPLATE
GGSCI (server1) 10> add trandata G001SCHEMA.CFG_REG_REPORT_RULES
GGSCI (server1) 11> add trandata G001SCHEMA.CMN_LOOKUP
GGSCI (server1) 13> add trandata G001SCHEMA.CMN_USER_LOGIN

• Configure the source database to extract the additional five tables and we can restart the corresponding extract groups

GGSCI (server1) 30> start extract *
GGSCI (server1) 31> info all
Program Status Group Lag Time Since Chkpt
MANAGER RUNNING
EXTRACT RUNNING DPG001 00:23:02 00:00:03
EXTRACT RUNNING G001 00:00:00 00:00:06

Initial Load from the new tables

As the addtional tables from the source database PROD1 are extracted, the initial load (expdp/impdp) from the data can be started on the target database REP1.
Attention, we have to start the export for a specific transaction point (SCN) , in order to begin the replication target database from the same SCN.

Thus we have to going to read the current_scn from the source database for the synchronization (explained later).

SQL> select current_scn from v$database;

CURRENT_SCN
-----------
 1657153626

• Create an expdp parfile with the above selected SCN and the additional tables to replicate

oracle@server1:~/app/oracle/admin/PROD1/create/goldengate/add_tables [PROD1] cat expdp_additional_tables.par

flashback_scn=1657153626
SCHEMAS=G001SCHEMA
DUMPFILE=export_tables_G001SCHEMA.dmp
LOGFILE=export_tables_G001SCHEMA.log
INCLUDE=TABLE:"IN('CFG_ADV_COND','CFG_NARRATIVE_TEMPLATE','CFG_REG_REPORT_RULES','CMN_LOOKUP','CMN_USER_LOGIN')"
DIRECTORY=DATAPUMPDIR

• Start the export from the above tables as user system

oracle@server1:~/app/oracle/admin/PROD1/create/goldengate/ [PROD1] expdp parfile=expdp_additional_tables.par

Username: system
Password:

Connected to: Oracle Database 10g Enterprise Edition Release 10.2.0.5.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
FLASHBACK automatically enabled to preserve database integrity.
Starting "SYSTEM"."SYS_EXPORT_SCHEMA_01": system/******** parfile=expdp_additional_tables.par
Estimate in progress using BLOCKS method...
Processing object type SCHEMA_EXPORT/TABLE/TABLE_DATA
Total estimation using BLOCKS method: 21.25 MB
Processing object type SCHEMA_EXPORT/TABLE/TABLE
Processing object type SCHEMA_EXPORT/TABLE/GRANT/OWNER_GRANT/OBJECT_GRANT
Processing object type SCHEMA_EXPORT/TABLE/INDEX/INDEX
Processing object type SCHEMA_EXPORT/TABLE/CONSTRAINT/CONSTRAINT
Processing object type SCHEMA_EXPORT/TABLE/INDEX/STATISTICS/INDEX_STATISTICS
Processing object type SCHEMA_EXPORT/TABLE/CONSTRAINT/REF_CONSTRAINT
Processing object type SCHEMA_EXPORT/TABLE/STATISTICS/TABLE_STATISTICS
. . exported "G001SCHEMA"."CMN_USER_LOGIN" 8.735 MB 286163 rows
. . exported "G001SCHEMA"."CFG_ADV_COND" 1.523 MB 1617 rows
. . exported "G001SCHEMA"."CFG_REG_REPORT_RULES" 100.5 KB 714 rows
. . exported "G001SCHEMA"."CFG_NARRATIVE_TEMPLATE" 9.789 KB 85 rows
. . exported "G001SCHEMA"."CMN_LOOKUP" 6.585 KB 4 rows
Master table "SYSTEM"."SYS_EXPORT_SCHEMA_01" successfully loaded/unloaded
******************************************************************************
Dump file set for SYSTEM.SYS_EXPORT_SCHEMA_01 is:
 /u00/app/oracle/admin/PROD1/dmp/export_ARGUS_tables_G001SCHEMA.dmp
Job "SYSTEM"."SYS_EXPORT_SCHEMA_01" successfully completed at 11:01:22

• Copy the dumpfile from server1 to server2

oracle@server1:~/app/oracle/admin/PROD1/create/goldengate/ [PROD1] 
scp /u00/app/oracle/admin/PROD1/dmp/export_tables_G001SCHEMA.dmp oracle@server2:~/app/oracle/admin/REP1/dmp/
export_tables_G001SCHEMA.dmp 100% 11MB 10.7MB/s 00:00

• Create an impdp parfile in order to load the tables from the above created dumpfile

Take care: the correct REMAP and EXCLUDE parameter must be configured based on your environment requirement

oracle@server2:~/app/oracle/admin/REP1/create/goldengate/add_tables/ [REP1] cat impdp_PROD1_tables.par

DUMPFILE=export_tables_G001SCHEMA.dmp
LOGFILE=import_tables_G001SCHEMA.log
REMAP_SCHEMA=G001SCHEMA:G001_PROD1
REMAP_TABLESPACE=DATA_01:PROD1_DATA
REMAP_TABLESPACE=DATA_02:PROD1_DATA
REMAP_TABLESPACE=INDEX_01:PROD1_DATA
REMAP_TABLESPACE=INDEX_02:PROD1_DATA
DIRECTORY=DATAPUMPDIR
EXCLUDE=GRANT
EXCLUDE=CONSTRAINT
EXCLUDE=REF_CONSTRAINT

• Start the import with impdp as user system using the above parameter file

oracle@server2:~/app/oracle/admin/REP1/create/goldengate/add_tables/ [REP1] impdp parfile=impdp_PROD1_tables.par

Username: system
Password:

Connected to: Oracle Database 11g Enterprise Edition Release 11.2.0.2.0 - 64bit Production
With the Partitioning, Oracle Label Security, OLAP, Data Mining,
Oracle Database Vault and Real Application Testing options
Master table "SYSTEM"."SYS_IMPORT_FULL_01" successfully loaded/unloaded
Starting "SYSTEM"."SYS_IMPORT_FULL_01": system/******** parfile=impdp_PROD1_tables.par
Processing object type SCHEMA_EXPORT/TABLE/TABLE
Processing object type SCHEMA_EXPORT/TABLE/TABLE_DATA
. . imported "G001_PROD1"."CMN_USER_LOGIN" 8.735 MB 286163 rows
. . imported "G001_PROD1"."CFG_ADV_COND" 1.523 MB 1617 rows
. . imported "G001_PROD1"."CFG_REG_REPORT_RULES" 100.5 KB 714 rows
. . imported "G001_PROD1"."CFG_NARRATIVE_TEMPLATE" 9.789 KB 85 rows
. . imported "G001_PROD1"."CMN_LOOKUP" 6.585 KB 4 rows
Processing object type SCHEMA_EXPORT/TABLE/INDEX/INDEX
Processing object type SCHEMA_EXPORT/TABLE/INDEX/STATISTICS/INDEX_STATISTICS
Processing object type SCHEMA_EXPORT/TABLE/STATISTICS/TABLE_STATISTICS
Job "SYSTEM"."SYS_IMPORT_FULL_01" successfully completed at 11:49:06

Add temporary the new tables to replication group

Now the new tables to replicate can be temporary add to the replication group PROD1 with the above selected SCN number, as starting point for the replication.

Attention: Within the GoldenGate configuration file we speak about CSN and not SCN, but both contains the same information (System Change Number).

oracle@server2:~/app/goldengate/ggs/11.1.1.1.0/[REP1] ggsci

GGSCI (server2)> edit params PROD1

REPLICAT PROD1
ASSUMETARGETDEFS
USERID goldengate, PASSWORD *****
DISCARDFILE /u00/app/goldengate/ggs/11.1.1.1.0/discard/PROD1_discard.txt, append, megabytes 10
MAP G001schema.CFG_ADV_COND ,TARGET G001_PROD1.CFG_ADV_COND ,FILTER ( @GETENV ("TRANSACTION", "CSN") > 1657153626);
MAP G001schema.CFG_NARRATIVE_TEMPLATE ,TARGET G001_PROD1.CFG_NARRATIVE_TEMPLATE ,FILTER ( @GETENV ("TRANSACTION", "CSN") > 1657153626);
MAP G001schema.CFG_REG_REPORT_RULES ,TARGET G001_PROD1.CFG_REG_REPORT_RULES ,FILTER ( @GETENV ("TRANSACTION", "CSN") > 1657153626);
MAP G001schema.CMN_LOOKUP ,TARGET G001_PROD1.CMN_LOOKUP ,FILTER ( @GETENV ("TRANSACTION", "CSN") > 1657153626);
MAP G001schema.CMN_USER_LOGIN ,TARGET G001_PROD1.CMN_USER_LOGIN ,FILTER ( @GETENV ("TRANSACTION", "CSN") > 1657153626);
MAP G001schema.*, TARGET G001_PROD1.*

• Start the replication group PROD1 with the new added tables

GGSCI (server2) 2> start replicat PROD1

Sending START request to MANAGER ...
REPLICAT PROD1 starting

• Check the configuration until the both databases are synchronised

GGSCI (server2) 1> info all
Program Status Group Lag Time Since Chkpt
MANAGER RUNNING
REPLICAT RUNNING PROD1 00:00:00 00:00:00

Remove the temporary added new tables from the replication group file

As both databases are again synchronized including the new tables, the temporary added filter can be removed from the replication group file.

• Stop the replication group PROD1

GGSCI (server2) 4> stop replicat PROD1

• Set back the original extraction configuration file for the froup PROD, without the filters for the new files

oracle@server2:~/app/goldengate/ggs/11.1.1.1.0/[REP1] ggsci

GGSCI (server2) 2> edit params PROD1

REPLICAT PROD1
ASSUMETARGETDEFS
USERID goldengate, PASSWORD *****
DISCARDFILE /u00/app/goldengate/ggs/11.1.1.1.0/discard/PROD1_discard.txt, append, megabytes 10
MAP G001schema.*, TARGET G001_PROD1.*

• Restart the replication

GGSCI (server2) 3> start REPLICAT PROD1
GGSCI (server2) 4> info all

Program Status Group Lag Time Since Chkpt
MANAGER RUNNING
REPLICAT RUNNING PROD1 00:00:00 00:00:00

Conclusion

The integration of additional tables into a existing GoldenGate replication environment is not very complex, but the steps order must be exactly and carefully followed. Have fun with Oracle Goldengate!

L’article Adding new tables to an existing Oracle GoldenGate replication est apparu en premier sur dbi Blog.

In his workshop, Jonathan Lewis covered an impressive amount of Oracle optimizer know how: from the fundamentals of the Cost Based Optimizer (CBO) to the latest 11.2 Oracle Optimizer features. Useless to say that we were all amazed from Jonathan’s expertise.

Jonathan provided an impressive insight into the Oracle Optimizer which really helped me to better grasp the decisions of the Cost Based Optimizer (CBO). Understanding how and why nested loops or hash joins are used is a black box if you are not aware of the underlying “maths”. This kind of information cleary was a big added value of Jonathan’s workshop. He also gave us lots of practical and very funny examples of his experience at customer sites.

As most of you know, Jonathan Lewis is a well known figure in the Oracle community and has also published several books (last publication in November 2011: Oracle Core: Apress, ISBN 978-1430239543). His blog really is a key Oracle resource (http://jonathanlewis.wordpress.com/) and he also contributes to several other blogs, newsgroups, forums, and Oracle User Group magazines.

Again, Jonathan, thanks a lot for your great workshop!

L’article Jonathan Lewis @ dbi services: Oracle Optimizer workshop est apparu en premier sur dbi Blog.

Analysis

I began checking why the diagnostic directory was growing so fast. And I found out that all 11g databases – the new one and all the other ones – reported the message below each second in the database alert.log file.

WARNING: Oracle executable binary mismatch detected.
Binary of new process does not match binary which started instance
issue alter system set "_disable_image_check" = true to disable these messages

This problem occured because during the database upgrade procedure, DBUA recompiles the Oracle binary executable. Thus all databases using this Oracle executable file send WARNINGS that the executable was recompiled.

oracle@chbsslcmdbdb1:~/app/oracle/product/11.2.0.2/bin/ [DBACT411] ls -lrt
 ....
 -rwxr-xr-x 1 oracle dba      3771 Oct 28  2010 emagentdeploy.pl
 -rwxr-xr-x 1 oracle dba      6299 Oct 28  2010 dbca
 -rwxr-xr-x 1 oracle dba      3552 Oct 28  2010 db2gc
 -rwsr-s--x 1 oracle dba 228886356 Jun  9 09:59 oracle

How to fix this problem

A restart from all affected database must be done.

My recommendation

When you upgrade an 11g database on a server where you still have a 11g database running that you can not restart: Install a new ORACLE_HOME and this problem will only affect your new database which can be fixed with a simple restart of the database.

L’article Take care while upgrading your Oracle DB with “DBUA” est apparu en premier sur dbi Blog.