ZFS for Solaris is around for several years now (since 2015). But there is also a project called OpenZFS which makes ZFS available on other operating systems. For Linux the announcement for ZFS being production ready was back in 2013. So why not run PostgreSQL on it? ZFS provides many cool features including compression, snapshots and build in volume management. Lets give it a try and do an initial setup. More details will follow in separate posts.

As usual I am running a CentOS 7 VM for my tests:

[root@centos7 ~] lsb_release -a
LSB Version:	:core-4.1-amd64:core-4.1-noarch:cxx-4.1-amd64:cxx-4.1-noarch:desktop-4.1-amd64:desktop-4.1-noarch:languages-4.1-amd64:languages-4.1-noarch:printing-4.1-amd64:printing-4.1-noarch
Distributor ID:	CentOS
Description:	CentOS Linux release 7.2.1511 (Core) 
Release:	7.2.1511
Codename:	Core

There is a dedicated website for ZFS on Linux where you can find the instructions on how to install it for various distributions. The instruction for CentOS/RHEL are quite easy. Download the repo files:

[root@centos7 ~] yum install http://download.zfsonlinux.org/epel/zfs-release$(rpm -E %dist).noarch.rpm
Loaded plugins: fastestmirror
zfs-release.el7.centos.noarch.rpm                                                                    | 5.0 kB  00:00:00     
Examining /var/tmp/yum-root-Uv79vc/zfs-release.el7.centos.noarch.rpm: zfs-release-1-3.el7.centos.noarch
Marking /var/tmp/yum-root-Uv79vc/zfs-release.el7.centos.noarch.rpm to be installed
Resolving Dependencies
--> Running transaction check
---> Package zfs-release.noarch 0:1-3.el7.centos will be installed
--> Finished Dependency Resolution

Dependencies Resolved

============================================================================================================================
 Package                  Arch                Version                     Repository                                   Size
============================================================================================================================
Installing:
 zfs-release              noarch              1-3.el7.centos              /zfs-release.el7.centos.noarch              2.9 k

Transaction Summary
============================================================================================================================
Install  1 Package

Total size: 2.9 k
Installed size: 2.9 k
Is this ok [y/d/N]: y
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : zfs-release-1-3.el7.centos.noarch                                                                        1/1 
  Verifying  : zfs-release-1-3.el7.centos.noarch                                                                        1/1 

Installed:
  zfs-release.noarch 0:1-3.el7.centos                                                                                       

Complete!

[root@centos7 ~] gpg --quiet --with-fingerprint /etc/pki/rpm-gpg/RPM-GPG-KEY-zfsonlinux
gpg: new configuration file `/root/.gnupg/gpg.conf' created
gpg: WARNING: options in `/root/.gnupg/gpg.conf' are not yet active during this run
pub  2048R/F14AB620 2013-03-21 ZFS on Linux 
      Key fingerprint = C93A FFFD 9F3F 7B03 C310  CEB6 A9D5 A1C0 F14A B620
sub  2048R/99685629 2013-03-21

For the next step it depends if you want to go with DKMS or kABI-tracking kmod. I’ll go with kABI-tracking kmod and therefore will disable the DKMS repository and enable the kmod repository:

[root@centos7 ~] cat /etc/yum.repos.d/zfs.repo 
[zfs]
name=ZFS on Linux for EL7 - dkms
baseurl=http://download.zfsonlinux.org/epel/7/$basearch/
enabled=0
metadata_expire=7d
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-zfsonlinux

[zfs-kmod]
name=ZFS on Linux for EL7 - kmod
baseurl=http://download.zfsonlinux.org/epel/7/kmod/$basearch/
enabled=1
metadata_expire=7d
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-zfsonlinux

[zfs-source]
name=ZFS on Linux for EL7 - Source
baseurl=http://download.zfsonlinux.org/epel/7/SRPMS/
enabled=0
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-zfsonlinux

[zfs-testing]
name=ZFS on Linux for EL7 - dkms - Testing
baseurl=http://download.zfsonlinux.org/epel-testing/7/$basearch/
enabled=0
metadata_expire=7d
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-zfsonlinux

[zfs-testing-kmod]
name=ZFS on Linux for EL7 - kmod - Testing
baseurl=http://download.zfsonlinux.org/epel-testing/7/kmod/$basearch/
enabled=0
metadata_expire=7d
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-zfsonlinux

[zfs-testing-source]
name=ZFS on Linux for EL7 - Testing Source
baseurl=http://download.zfsonlinux.org/epel-testing/7/SRPMS/
enabled=0
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-zfsonlinux
[root@centos7 ~] 

Installing ZFS from here on is just a matter of using yum:

[root@centos7 ~] yum install zfs
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: mirror.spreitzer.ch
 * extras: mirror.spreitzer.ch
 * updates: mirror.de.leaseweb.net
zfs-kmod/x86_64/primary_db                                                                           | 231 kB  00:00:01     
Resolving Dependencies
--> Running transaction check
---> Package zfs.x86_64 0:0.6.5.8-1.el7.centos will be installed
--> Processing Dependency: zfs-kmod = 0.6.5.8 for package: zfs-0.6.5.8-1.el7.centos.x86_64
--> Processing Dependency: spl = 0.6.5.8 for package: zfs-0.6.5.8-1.el7.centos.x86_64
--> Processing Dependency: libzpool2 = 0.6.5.8 for package: zfs-0.6.5.8-1.el7.centos.x86_64
--> Processing Dependency: libzfs2 = 0.6.5.8 for package: zfs-0.6.5.8-1.el7.centos.x86_64
--> Processing Dependency: libuutil1 = 0.6.5.8 for package: zfs-0.6.5.8-1.el7.centos.x86_64
--> Processing Dependency: libnvpair1 = 0.6.5.8 for package: zfs-0.6.5.8-1.el7.centos.x86_64
--> Processing Dependency: libzpool.so.2()(64bit) for package: zfs-0.6.5.8-1.el7.centos.x86_64
--> Processing Dependency: libzfs_core.so.1()(64bit) for package: zfs-0.6.5.8-1.el7.centos.x86_64
--> Processing Dependency: libzfs.so.2()(64bit) for package: zfs-0.6.5.8-1.el7.centos.x86_64
--> Processing Dependency: libuutil.so.1()(64bit) for package: zfs-0.6.5.8-1.el7.centos.x86_64
--> Processing Dependency: libnvpair.so.1()(64bit) for package: zfs-0.6.5.8-1.el7.centos.x86_64
--> Running transaction check
---> Package kmod-zfs.x86_64 0:0.6.5.8-1.el7.centos will be installed
--> Processing Dependency: spl-kmod for package: kmod-zfs-0.6.5.8-1.el7.centos.x86_64
---> Package libnvpair1.x86_64 0:0.6.5.8-1.el7.centos will be installed
---> Package libuutil1.x86_64 0:0.6.5.8-1.el7.centos will be installed
---> Package libzfs2.x86_64 0:0.6.5.8-1.el7.centos will be installed
---> Package libzpool2.x86_64 0:0.6.5.8-1.el7.centos will be installed
---> Package spl.x86_64 0:0.6.5.8-1.el7.centos will be installed
--> Running transaction check
---> Package kmod-spl.x86_64 0:0.6.5.8-1.el7.centos will be installed
--> Finished Dependency Resolution

Dependencies Resolved

============================================================================================================================
 Package                     Arch                    Version                                Repository                 Size
============================================================================================================================
Installing:
 zfs                         x86_64                  0.6.5.8-1.el7.centos                   zfs-kmod                  334 k
Installing for dependencies:
 kmod-spl                    x86_64                  0.6.5.8-1.el7.centos                   zfs-kmod                  110 k
 kmod-zfs                    x86_64                  0.6.5.8-1.el7.centos                   zfs-kmod                  665 k
 libnvpair1                  x86_64                  0.6.5.8-1.el7.centos                   zfs-kmod                   35 k
 libuutil1                   x86_64                  0.6.5.8-1.el7.centos                   zfs-kmod                   41 k
 libzfs2                     x86_64                  0.6.5.8-1.el7.centos                   zfs-kmod                  123 k
 libzpool2                   x86_64                  0.6.5.8-1.el7.centos                   zfs-kmod                  423 k
 spl                         x86_64                  0.6.5.8-1.el7.centos                   zfs-kmod                   29 k

Transaction Summary
============================================================================================================================
Install  1 Package (+7 Dependent packages)

Total download size: 1.7 M
Installed size: 5.7 M
Is this ok [y/d/N]: y
Downloading packages:
(1/8): kmod-spl-0.6.5.8-1.el7.centos.x86_64.rpm                                                      | 110 kB  00:00:01     
(2/8): libnvpair1-0.6.5.8-1.el7.centos.x86_64.rpm                                                    |  35 kB  00:00:00     
(3/8): libuutil1-0.6.5.8-1.el7.centos.x86_64.rpm                                                     |  41 kB  00:00:00     
(4/8): kmod-zfs-0.6.5.8-1.el7.centos.x86_64.rpm                                                      | 665 kB  00:00:02     
(5/8): libzfs2-0.6.5.8-1.el7.centos.x86_64.rpm                                                       | 123 kB  00:00:00     
(6/8): libzpool2-0.6.5.8-1.el7.centos.x86_64.rpm                                                     | 423 kB  00:00:00     
(7/8): spl-0.6.5.8-1.el7.centos.x86_64.rpm                                                           |  29 kB  00:00:00     
(8/8): zfs-0.6.5.8-1.el7.centos.x86_64.rpm                                                           | 334 kB  00:00:00     
----------------------------------------------------------------------------------------------------------------------------
Total                                                                                       513 kB/s | 1.7 MB  00:00:03     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : libuutil1-0.6.5.8-1.el7.centos.x86_64                                                                    1/8 
  Installing : libnvpair1-0.6.5.8-1.el7.centos.x86_64                                                                   2/8 
  Installing : libzpool2-0.6.5.8-1.el7.centos.x86_64                                                                    3/8 
  Installing : kmod-spl-0.6.5.8-1.el7.centos.x86_64                                                                     4/8 
  Installing : spl-0.6.5.8-1.el7.centos.x86_64                                                                          5/8 
  Installing : libzfs2-0.6.5.8-1.el7.centos.x86_64                                                                      6/8 
  Installing : kmod-zfs-0.6.5.8-1.el7.centos.x86_64                                                                     7/8 
  Installing : zfs-0.6.5.8-1.el7.centos.x86_64                                                                          8/8 
  Verifying  : libnvpair1-0.6.5.8-1.el7.centos.x86_64                                                                   1/8 
  Verifying  : libzfs2-0.6.5.8-1.el7.centos.x86_64                                                                      2/8 
  Verifying  : zfs-0.6.5.8-1.el7.centos.x86_64                                                                          3/8 
  Verifying  : spl-0.6.5.8-1.el7.centos.x86_64                                                                          4/8 
  Verifying  : kmod-zfs-0.6.5.8-1.el7.centos.x86_64                                                                     5/8 
  Verifying  : libzpool2-0.6.5.8-1.el7.centos.x86_64                                                                    6/8 
  Verifying  : libuutil1-0.6.5.8-1.el7.centos.x86_64                                                                    7/8 
  Verifying  : kmod-spl-0.6.5.8-1.el7.centos.x86_64                                                                     8/8 

Installed:
  zfs.x86_64 0:0.6.5.8-1.el7.centos                                                                                         

Dependency Installed:
  kmod-spl.x86_64 0:0.6.5.8-1.el7.centos   kmod-zfs.x86_64 0:0.6.5.8-1.el7.centos  libnvpair1.x86_64 0:0.6.5.8-1.el7.centos 
  libuutil1.x86_64 0:0.6.5.8-1.el7.centos  libzfs2.x86_64 0:0.6.5.8-1.el7.centos   libzpool2.x86_64 0:0.6.5.8-1.el7.centos  
  spl.x86_64 0:0.6.5.8-1.el7.centos       

Complete!
[root@centos7 ~]

Be aware that the kernel modules are not loaded by default, so you have to do this on your own:

[root@centos7 ~] /sbin/modprobe zfs
Last login: Wed Sep 28 11:04:21 2016 from 192.168.22.1
[postgres@centos7 ~]$ lsmod | grep zfs
zfs                  2713912  0 
zunicode              331170  1 zfs
zavl                   15236  1 zfs
zcommon                55411  1 zfs
znvpair                93227  2 zfs,zcommon
spl                    92223  3 zfs,zcommon,znvpair
[root@centos7 ~] zfs list
no datasets available

For loading the modules automatically create a file under /etc/modules-load.d:

[root@centos7 ~] echo "zfs" > /etc/modules-load.d/zfs.conf
[root@centos7 ~] cat /etc/modules-load.d/zfs.conf
zfs

So far so good. Lets create a ZFS file system. I have two disks available for playing with ZFS (sdb and sdc):

[root@centos7 ~] ls -la /dev/sd*
brw-rw----. 1 root disk 8,  0 Sep 28 11:14 /dev/sda
brw-rw----. 1 root disk 8,  1 Sep 28 11:14 /dev/sda1
brw-rw----. 1 root disk 8,  2 Sep 28 11:14 /dev/sda2
brw-rw----. 1 root disk 8, 16 Sep 28 11:14 /dev/sdb
brw-rw----. 1 root disk 8, 32 Sep 28 11:14 /dev/sdc

The first thing you have to do is to create a new zfs pool (I don’t care about the warnings, that is why I use the “-f” option below):

[root@centos7 ~] zpool create pgpool mirror /dev/sdb /dev/sdc
invalid vdev specification
use '-f' to override the following errors:
/dev/sdb does not contain an EFI label but it may contain partition information in the MBR.
/dev/sdc does not contain an EFI label but it may contain partition information in the MBR.
[root@centos7 ~] zpool create pgpool mirror /dev/sdb /dev/sdc -f
[root@centos7 ~] zpool list
NAME     SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
pgpool  9.94G    65K  9.94G         -     0%     0%  1.00x  ONLINE  -
[root@centos7 ~] zpool status pgpool
  pool: pgpool
 state: ONLINE
  scan: none requested
config:

	NAME        STATE     READ WRITE CKSUM
	pgpool      ONLINE       0     0     0
	  mirror-0  ONLINE       0     0     0
	    sdb     ONLINE       0     0     0
	    sdc     ONLINE       0     0     0

errors: No known data errors

[root@centos7 ~] df -h
Filesystem               Size  Used Avail Use% Mounted on
/dev/mapper/centos-root   49G  1.7G   47G   4% /
devtmpfs                 235M     0  235M   0% /dev
tmpfs                    245M     0  245M   0% /dev/shm
tmpfs                    245M  4.3M  241M   2% /run
tmpfs                    245M     0  245M   0% /sys/fs/cgroup
/dev/sda1                497M  291M  206M  59% /boot
tmpfs                     49M     0   49M   0% /run/user/1000
pgpool                   9.7G     0  9.7G   0% /pgpool

What I did here is to create a mirrored pool over my two disks. The open zfs wiki has some performance tips for running PostgreSQL on ZFS as well as for other topics. Lets go with the recommendations:

[root@centos7 ~] zfs create pgpool/pgdata -o recordsize=8192
[root@centos7 ~] zfs set logbias=throughput pgpool/pgdata
[root@centos7 ~] zfs set primarycache=all pgpool/pgdata
[root@centos7 ~] zfs list
NAME            USED  AVAIL  REFER  MOUNTPOINT
pgpool           82K  9.63G  19.5K  /pgpool
pgpool/pgdata    19K  9.63G    19K  /pgpool/pgdata

My new ZFS file system is ready and already mounted, cool. Lets change the permissions and list all the properties:

[root@centos7 ~] chown postgres:postgres /pgpool/pgdata
[root@centos7 ~] zfs get all /pgpool/pgdata
NAME           PROPERTY              VALUE                  SOURCE
pgpool/pgdata  type                  filesystem             -
pgpool/pgdata  creation              Wed Sep 28 11:31 2016  -
pgpool/pgdata  used                  19K                    -
pgpool/pgdata  available             9.63G                  -
pgpool/pgdata  referenced            19K                    -
pgpool/pgdata  compressratio         1.00x                  -
pgpool/pgdata  mounted               yes                    -
pgpool/pgdata  quota                 none                   default
pgpool/pgdata  reservation           none                   default
pgpool/pgdata  recordsize            8K                     local
pgpool/pgdata  mountpoint            /pgpool/pgdata         default
pgpool/pgdata  sharenfs              off                    default
pgpool/pgdata  checksum              on                     default
pgpool/pgdata  compression           off                    default
pgpool/pgdata  atime                 on                     default
pgpool/pgdata  devices               on                     default
pgpool/pgdata  exec                  on                     default
pgpool/pgdata  setuid                on                     default
pgpool/pgdata  readonly              off                    default
pgpool/pgdata  zoned                 off                    default
pgpool/pgdata  snapdir               hidden                 default
pgpool/pgdata  aclinherit            restricted             default
pgpool/pgdata  canmount              on                     default
pgpool/pgdata  xattr                 on                     default
pgpool/pgdata  copies                1                      default
pgpool/pgdata  version               5                      -
pgpool/pgdata  utf8only              off                    -
pgpool/pgdata  normalization         none                   -
pgpool/pgdata  casesensitivity       sensitive              -
pgpool/pgdata  vscan                 off                    default
pgpool/pgdata  nbmand                off                    default
pgpool/pgdata  sharesmb              off                    default
pgpool/pgdata  refquota              none                   default
pgpool/pgdata  refreservation        none                   default
pgpool/pgdata  primarycache          all                    default
pgpool/pgdata  secondarycache        all                    default
pgpool/pgdata  usedbysnapshots       0                      -
pgpool/pgdata  usedbydataset         19K                    -
pgpool/pgdata  usedbychildren        0                      -
pgpool/pgdata  usedbyrefreservation  0                      -
pgpool/pgdata  logbias               throughput             local
pgpool/pgdata  dedup                 off                    default
pgpool/pgdata  mlslabel              none                   default
pgpool/pgdata  sync                  standard               default
pgpool/pgdata  refcompressratio      1.00x                  -
pgpool/pgdata  written               19K                    -
pgpool/pgdata  logicalused           9.50K                  -
pgpool/pgdata  logicalreferenced     9.50K                  -
pgpool/pgdata  filesystem_limit      none                   default
pgpool/pgdata  snapshot_limit        none                   default
pgpool/pgdata  filesystem_count      none                   default
pgpool/pgdata  snapshot_count        none                   default
pgpool/pgdata  snapdev               hidden                 default
pgpool/pgdata  acltype               off                    default
pgpool/pgdata  context               none                   default
pgpool/pgdata  fscontext             none                   default
pgpool/pgdata  defcontext            none                   default
pgpool/pgdata  rootcontext           none                   default
pgpool/pgdata  relatime              on                     temporary
pgpool/pgdata  redundant_metadata    all                    default
pgpool/pgdata  overlay               off                    default

Ready to deploy a PostgreSQL instance on it:

postgres@centos7:/home/postgres/ [pg954] initdb -D /pgpool/pgdata/
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locales
  COLLATE:  en_US.UTF-8
  CTYPE:    en_US.UTF-8
  MESSAGES: en_US.UTF-8
  MONETARY: de_CH.UTF-8
  NUMERIC:  de_CH.UTF-8
  TIME:     en_US.UTF-8
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /pgpool/pgdata ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting dynamic shared memory implementation ... posix
creating configuration files ... ok
creating template1 database in /pgpool/pgdata/base/1 ... ok
initializing pg_authid ... ok
initializing dependencies ... ok
creating system views ... ok
loading system objects' descriptions ... ok
creating collations ... ok
creating conversions ... ok
creating dictionaries ... ok
setting privileges on built-in objects ... ok
creating information schema ... ok
loading PL/pgSQL server-side language ... ok
vacuuming database template1 ... ok
copying template1 to template0 ... ok
copying template1 to postgres ... ok
syncing data to disk ... ok

WARNING: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.

Success. You can now start the database server using:

    pg_ctl -D /pgpool/pgdata/ -l logfile start

Startup:

postgres@centos7:/home/postgres/ [pg954] mkdir /pgpool/pgdata/pg_log
postgres@centos7:/home/postgres/ [pg954] sed -i 's/logging_collector = off/logging_collector = on/g' /pgpool/pgdata/postgresql.conf
postgres@centos7:/home/postgres/ [pg954] pg_ctl -D /pgpool/pgdata/ start
postgres@centos7:/home/postgres/ [pg954] psql postgres
psql (9.5.4 dbi services build)
Type "help" for help.

postgres=

Ready. Lets reboot and check if the ZFS file system is mounted automatically:

postgres@centos7:/home/postgres/ [pg954] df -h
Filesystem               Size  Used Avail Use% Mounted on
/dev/mapper/centos-root   49G  1.8G   47G   4% /
devtmpfs                 235M     0  235M   0% /dev
tmpfs                    245M     0  245M   0% /dev/shm
tmpfs                    245M  4.3M  241M   2% /run
tmpfs                    245M     0  245M   0% /sys/fs/cgroup
/dev/sda1                497M  291M  206M  59% /boot
tmpfs                     49M     0   49M   0% /run/user/1000
postgres@centos7:/home/postgres/ [pg954] lsmod | grep zfs
zfs                  2713912  0 
zunicode              331170  1 zfs
zavl                   15236  1 zfs
zcommon                55411  1 zfs
znvpair                93227  2 zfs,zcommon
spl                    92223  3 zfs,zcommon,znvpair

Gone. The kernel modules are loaded but the file system was not mounted. What to do?

[root@centos7 ~] zpool list
no pools available
[root@centos7 ~] zpool import pgpool
[root@centos7 ~] zpool list
NAME     SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
pgpool  9.94G  39.3M  9.90G         -     0%     0%  1.00x  ONLINE  -
[root@centos7 ~]# df -h
Filesystem               Size  Used Avail Use% Mounted on
/dev/mapper/centos-root   49G  1.8G   47G   4% /
devtmpfs                 235M     0  235M   0% /dev
tmpfs                    245M     0  245M   0% /dev/shm
tmpfs                    245M  4.3M  241M   2% /run
tmpfs                    245M     0  245M   0% /sys/fs/cgroup
/dev/sda1                497M  291M  206M  59% /boot
tmpfs                     49M     0   49M   0% /run/user/1000
pgpool                   9.6G     0  9.6G   0% /pgpool
pgpool/pgdata            9.7G   39M  9.6G   1% /pgpool/pgdata

Ok, how to auto mount?

[root@centos7 ~] systemctl enable zfs-mount
[root@centos7 ~] systemctl enable zfs-import-cache
[root@centos7 ~] reboot

I am not sure why this is necessary, should happen automatically.

PS: There is an interesting discussion about PostgreSQL on ZFS on the PostgreSQL performance mailing list currently.

Cet article Running PostgreSQL on ZFS on Linux est apparu en premier sur Blog dbi services.

J-1 before the next SQL Saturday 2016 in Portugal.

I usually promote this event in France but this time this is for SQL Server Portuguese community. I will have the chance to talk about SQL Server AlwaysOn and troubleshooting scenarios. It will be also a good opportunity to meet portuguese people like Niko Neugebauer (@NikoNeugebauer), André Batista (@klunkySQL) and many others.

Moreover, Porto is definitely a beautiful city and a good way to finish the summer vacations.

Hope to see you there!

Cet article SQLSaturday #546 Porto – Troubleshooting scenarios with availability groups est apparu en premier sur Blog dbi services.

In the last post we looked at how to get a ZFS file system up and running on a CentOS 7 host and how to enable the auto mount of the ZFS file systems. In this post we’ll look at two of the features ZFS provides: Snapshots and clones.

A ZFS snapshot is a read only copy of a file system. How can we benefit from that when it comes to PostgreSQL. There are several scenarios where this can be useful. Imagine you are developing an application and you want to test the deployment of a new release on top of a previous release. What you probably want to have is a production like PostgreSQL instance with lots of data for being able to test the upgrade path. In addition it would be great if you can revert in seconds and start from scratch just in case you run into troubles or you missed one important point in the upgrade scripts. Using ZFS snapshots you can have all of this. Lets see.

Currently my PostgreSQL instance from the last post does not contain any user data, so lets generate some:

postgres= create table my_app_table ( a int, b varchar(50) );
CREATE TABLE
postgres=# with aa as 
postgres-# ( select * 
postgres(#     from generate_series (1,1000000) a
postgres(# )
postgres-# insert into my_app_table
postgres-# select aa.a, md5(aa.a::varchar)
postgres-#   from aa;
INSERT 0 1000000

This is the release we want to test our upgrade scripts from so lets create a snapshot of the current state of our instance:

[root@centos7 ~] zfs snapshot pgpool/pgdata@baserelease
[root@centos7 ~] zfs list -t snapshot
NAME                        USED  AVAIL  REFER  MOUNTPOINT
pgpool/pgdata@baserelease  16.6M      -   202M  -

The “@baserelease” is the name of the snapshot or to be correct everything after the “@” is the name of the snapshot.

Are you worried about consistency? This should not be an issue as PostgreSQL fsyncs the WAL so the instance should just start, apply all the wal records which are missing from the data files and you’re fine. Anyway, this is a scenario for testing: So as long as you have a consistent starting point you are fine.

A simple upgrade script could be:

postgres=# alter table my_app_table add column c date;
ALTER TABLE
postgres=# update my_app_table set c = now();
UPDATE 1000000

What happened to the snapshot?

[root@centos7 ~] zfs list -t snapshot
NAME                        USED  AVAIL  REFER  MOUNTPOINT
pgpool/pgdata@baserelease  78.3M      -   202M  -

As soon as you modify data the snapshot will grow, no surprise.

So you did run your tests and discovered some things you could improve and once you improved what you wanted you want to start from the same point again. When having a snapshot this is quite easy, just revert to the snapshot. Of course you’ll need to stop your PostgreSQL instance first:

postgres@centos7:/home/postgres/ [PG1] pg_ctl stop -D /pgpool/pgdata/ -m fast
waiting for server to shut down....LOG:  received fast shutdown request
LOG:  aborting any active transactions
LOG:  autovacuum launcher shutting down
LOG:  shutting down
 done
server stopped

As soon as the instance is down the snapshot can be reverted:

[root@centos7 ~] zfs rollback pgpool/pgdata@baserelease
[root@centos7 ~] zfs list -t snapshot
NAME                        USED  AVAIL  REFER  MOUNTPOINT
pgpool/pgdata@baserelease     1K      -   202M  -

When you check the data after you started the instance again it is exactly as it was before:

postgres@centos7:/home/postgres/ [PG1] pg_ctl start -D /pgpool/pgdata/
postgres@centos7:/home/postgres/ [PG1] LOG:  database system was not properly shut down; automatic recovery in progress
postgres@centos7:/home/postgres/ [PG1] psql postgres
psql (9.5.4 dbi services build)
Type "help" for help.

postgres= \d my_app_table
        Table "public.my_app_table"
 Column |         Type          | Modifiers 
--------+-----------------------+-----------
 a      | integer               | 
 b      | character varying(50) | 

Notice the message about the automatic recovery, that is when the wal is replayed. Now you can just start your upgrade script again, revert in case of issues, start again, revert again, and so on.

Another use case: Rapid cloning of PostgreSQL instances (clones are writable, snapshots not). How does that work? This is where clones come into the game. For being able to clone you need a snapshot as clones depend on snapshots. Another thing to keep in mind is that you can not delete a snapshot when you have a clone still sitting on top of it. Lets see how it works:

As said we need a snapshot:

[root@centos7 ~] zfs snapshot pgpool/pgdata@clonebase

On top of this snapshot we can now create a clone:

[root@centos7 ~] zfs create pgpool/clones
[root@centos7 ~] zfs clone pgpool/pgdata@clonebase pgpool/clones/1
[root@centos7 ~] zfs list
NAME              USED  AVAIL  REFER  MOUNTPOINT
pgpool            170M  9.46G    21K  /pgpool
pgpool/clones    20.5K  9.46G  19.5K  /pgpool/clones
pgpool/clones/1     1K  9.46G   169M  /pgpool/clones/1
pgpool/pgdata     170M  9.46G   169M  /pgpool/pgdata

Using the new clone we bring up another PostgreSQL instance in seconds, containing the exact data from the source of the clone:

postgres@centos7:/home/postgres/ [PG1] rm /pgpool/clones/1/*.pid
postgres@centos7:/home/postgres/ [PG1] sed -i 's/#port = 5432/port=5433/g' /pgpool/clones/1/postgresql.conf
postgres@centos7:/home/postgres/ [PG1] pg_ctl start -D /pgpool/clones/1/
postgres@centos7:/home/postgres/ [PG1] psql -p 5433 postgres
psql (9.5.4 dbi services build)
Type "help" for help.

postgres=

Quite cool and easy.

Conclusion: I am not sure if I’d use ZFS for production databases on Linux because I have not tested enough. But for development and testing purposes there are quite a few benefits such as snapshots and cloning. This can simply your processes a lot. You could even use snapshots and clones as a basis for your backups although I’d prefer barman or bart.

PS: To clean up:

[root@centos7 ~] zfs destroy pgpool/clones/1
[root@centos7 ~] zfs destroy pgpool/clones

Cet article Running PostgreSQL on ZFS on Linux – Fun with snapshots and clones est apparu en premier sur Blog dbi services.

A lot of new features popped up with RMAN 12c, like  Expansion of Multi-section support, or Simplified cross-platform migration and many many more. However, in this post I would like to highlight a small, but very helpful new feature which I demonstrated during my session at the SOUG day.

The automatic datafile creation

In earlier versions of Oracle, you might run into situations where you create a new tablespace, and some objects into it, and then, all of a sudden, the DB crashes or the datafile is lost, without ever being backed up.

This is where the follwing command comes into play:

alter database create datafile <NAME>;

However, in Oracle 12c, this is done automatically for you, in case you use RMAN.

Lets create a tablespace and afterwards resize the datafile.

SQL> create tablespace dbi datafile '/home/oracle/rman/dbi01.dbf' size 16M;

Tablespace created.

SQL> alter database datafile '/home/oracle/rman/dbi01.dbf' resize 32M;

Database altered.

Now lets create a table in the new tablespace.

SQL> create table t_dbi tablespace dbi as select * from dba_objects;

Table created.

SQL> select count(*) from t_dbi;

  COUNT(*)
----------
    120130

Afterwards, we simulate an error and then check what Oracle says.

$ echo destroy > /home/oracle/rman/dbi01.dbf

SQL> alter system flush buffer_cache; 

System altered. 

SQL> select count(*) from t_dbi; 
select count(*) from t_dbi 
* 
ERROR at line 1: 
ORA-03135: connection lost contact Process ID: 25538 
Session ID: 387 Serial number: 55345

From the alert.log

Errors in file /u00/app/oracle/diag/rdbms/ocm121/OCM121/trace/OCM121_ora_25904.trc:
ORA-01157: cannot identify/lock data file 11 - see DBWR trace file
ORA-01110: data file 11: '/home/oracle/rman/dbi01.dbf'

Ok. Datafile is gone, and we have no backup. Lets do a preview to see, how Oracle would restore the datafile.

RMAN> restore ( datafile 11 ) preview;

Starting restore at 30-SEP-2016 15:46:27
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=355 device type=DISK
allocated channel: ORA_DISK_2
channel ORA_DISK_2: SID=126 device type=DISK

datafile 11 will be created automatically during restore operation
using channel ORA_DISK_1
using channel ORA_DISK_2

List of Archived Log Copies for database with db_unique_name OCM121
=====================================================================

Key     Thrd Seq     S Low Time
------- ---- ------- - --------------------
519     1    529     A 30-SEP-2016 12:23:00
        Name: +FRA/OCM121/ARCHIVELOG/2016_09_30/thread_1_seq_529.454.923931563

recovery will be done up to SCN 7346834
Media recovery start SCN is 7346834
Recovery must be done beyond SCN 7346834 to clear datafile fuzziness
Finished restore at 30-SEP-2016 15:46:28

As you can see in the preview output, Oracle will create the datafile automatically for us. Ok. Lets try it.

RMAN> restore ( datafile 11 );

Starting restore at 30-SEP-2016 15:48:29
using channel ORA_DISK_1
using channel ORA_DISK_2

creating datafile file number=11 name=/home/oracle/rman/dbi01.dbf
restore not done; all files read only, offline, or already restored
Finished restore at 30-SEP-2016 15:48:31

RMAN> recover datafile 11;

Starting recover at 30-SEP-2016 15:49:17
using channel ORA_DISK_1
using channel ORA_DISK_2

starting media recovery
media recovery complete, elapsed time: 00:00:02

Finished recover at 30-SEP-2016 15:49:20

RMAN> alter database datafile 11 online;

Statement processed

Ready. We can now access our table again without running the “alter database create datafile” command during the restore. And the resize to 32M was also done for us. However, the resize came during the recovery part.

$ ls -l dbi01.dbf
-rw-r----- 1 oracle asmadmin 33562624 Sep 30 15:52 dbi01.dbf

From my point of view, a small, but quite helpful feature.

One last remark. I have done my demo with Oracle 12cR1 with July 2016 PSU. In that version, the RMAN PREVIEW command has a bug, which says that you cannot recover into a consistent state, even if you can.  After applying the following OnOff patch on top of July PSU 2016, the RMAN PREVIEW command works as expected.

Patch 20315311: RMAN-5119: RECOVERY CAN NOT BE DONE TO A CONSISTENT STATE

Cheers,

William

Cet article Oracle 12c and RMAN automatic datafile creation est apparu en premier sur Blog dbi services.

In the last posts in this little series we looked at how to get a ZFS file system up and running on a CentOS 7 host and how snapshots and clones can be used to simply processes such as testing and cloning PostgreSQL instances. In this post we’ll look at another feature of zfs: Compression.

The current status of my ZFS file systems is:

[root@centos7 ~] zfs list
NAME            USED  AVAIL  REFER  MOUNTPOINT
pgpool          170M  9.46G  20.5K  /pgpool
pgpool/pgdata   169M  9.46G   169M  /pgpool/pgdata

To check if compression is enabled:

[root@centos7 ~] zfs get compression pgpool/pgdata
NAME           PROPERTY     VALUE     SOURCE
pgpool/pgdata  compression  off       default

Lets create another file system and enable compression for it:

[root@centos7 ~] zfs create pgpool/pgdatacompressed
[root@centos7 ~] zfs list
NAME                      USED  AVAIL  REFER  MOUNTPOINT
pgpool                    170M  9.46G  20.5K  /pgpool
pgpool/pgdata             169M  9.46G   169M  /pgpool/pgdata
pgpool/pgdatacompressed    19K  9.46G    19K  /pgpool/pgdatacompressed
[root@centos7 ~] zfs get compression pgpool/pgdatacompressed
NAME                     PROPERTY     VALUE     SOURCE
pgpool/pgdatacompressed  compression  off       default
[root@centos7 ~] zfs set compression=on pgpool/pgdatacompressed
[root@centos7 ~] zfs get compression pgpool/pgdatacompressed
NAME                     PROPERTY     VALUE     SOURCE
pgpool/pgdatacompressed  compression  on        local

You can ask zfs to report the compression ratio for a file system:

[root@centos7 ~] zfs get compressratio pgpool/pgdatacompressed
NAME                     PROPERTY       VALUE  SOURCE
pgpool/pgdatacompressed  compressratio  1.00x  -
[root@centos7 ~] chown postgres:postgres /pgpool/pgdatacompressed/

The ratio is 1 which is because we do not have any data yet. Lets copy the PostgreSQL cluster from the uncompressed file system into our new compressed file system:

postgres@centos7:/home/postgres/ [PG1] cp -pr /pgpool/pgdata/* /pgpool/pgdatacompressed/
postgres@centos7:/home/postgres/ [PG1] ls -l /pgpool/pgdatacompressed/
total 30
drwx------. 6 postgres postgres     6 Sep 29 14:00 base
drwx------. 2 postgres postgres    54 Sep 29 14:27 global
drwx------. 2 postgres postgres     3 Sep 28 15:11 pg_clog
drwx------. 2 postgres postgres     2 Sep 28 15:11 pg_commit_ts
drwx------. 2 postgres postgres     2 Sep 28 15:11 pg_dynshmem
-rw-------. 1 postgres postgres  4468 Sep 28 15:11 pg_hba.conf
-rw-------. 1 postgres postgres  1636 Sep 28 15:11 pg_ident.conf
drwxr-xr-x. 2 postgres postgres     2 Sep 28 15:11 pg_log
drwx------. 4 postgres postgres     4 Sep 28 15:11 pg_logical
drwx------. 4 postgres postgres     4 Sep 28 15:11 pg_multixact
drwx------. 2 postgres postgres     3 Sep 29 14:27 pg_notify
drwx------. 2 postgres postgres     2 Sep 28 15:11 pg_replslot
drwx------. 2 postgres postgres     2 Sep 28 15:11 pg_serial
drwx------. 2 postgres postgres     2 Sep 28 15:11 pg_snapshots
drwx------. 2 postgres postgres     5 Sep 29 14:46 pg_stat
drwx------. 2 postgres postgres     2 Sep 29 14:46 pg_stat_tmp
drwx------. 2 postgres postgres     3 Sep 28 15:11 pg_subtrans
drwx------. 2 postgres postgres     2 Sep 28 15:11 pg_tblspc
drwx------. 2 postgres postgres     2 Sep 28 15:11 pg_twophase
-rw-------. 1 postgres postgres     4 Sep 28 15:11 PG_VERSION
drwx------. 3 postgres postgres     8 Sep 29 14:26 pg_xlog
-rw-------. 1 postgres postgres    88 Sep 28 15:11 postgresql.auto.conf
-rw-------. 1 postgres postgres 21270 Sep 28 15:11 postgresql.conf
-rw-------. 1 postgres postgres    69 Sep 29 14:27 postmaster.opts

We already should see a difference, shouldn’t we?

postgres@centos7:/home/postgres/ [PG1] df -h | grep pgdata
pgpool/pgdata            9.6G  170M  9.4G   2% /pgpool/pgdata
pgpool/pgdatacompressed  9.5G   82M  9.4G   1% /pgpool/pgdatacompressed

Not bad, less than half of the size. We should see another compression ratio than 1 now:

[root@centos7 ~] zfs get compressratio pgpool/pgdatacompressed
NAME                     PROPERTY       VALUE  SOURCE
pgpool/pgdatacompressed  compressratio  1.93x  -

Lets generate some data in our two PostgreSQL instances and check the time it takes as well as the size of the file systems afterwards. As in the last post the second instance just gets a different port, everything else is identical:

postgres@centos7:/home/postgres/ [PG1] pg_ctl start -D /pgpool/pgdata
postgres@centos7:/home/postgres/ [PG1] sed -i 's/#port = 5432/port=5433/g' /pgpool/pgdatacompressed/postgresql.conf
postgres@centos7:/home/postgres/ [PG1] FATAL:  data directory "/pgpool/pgdatacompressed" has group or world access
postgres@centos7:/home/postgres/ [PG1] chmod o-rwx,g-rwx /pgpool/pgdatacompressed/
postgres@centos7:/home/postgres/ [PG1] pg_ctl start -D /pgpool/pgdatacompressed/

This is the script to generate some data:

\timing
\c postgres
drop database if exists dataload;
create database dataload;
\c dataload
create table dataload ( a bigint
                      , b varchar(100)
                      , c timestamp
                      );
with 
  data_generator_num as
     ( select *
         from generate_series ( 1
                              , 1000000 ) nums
     ) 
insert into dataload
select data_generator_num.nums
     , md5(data_generator_num.nums::varchar)
     , current_date+data_generator_num.nums
 from data_generator_num;

I will run the script two times on each instance. For the instance on the uncompressed file system:

-- FIRST RUN
postgres=# \i generate_data.sql
Timing is on.
You are now connected to database "postgres" as user "postgres".
DROP DATABASE
Time: 720.626 ms
CREATE DATABASE
Time: 4631.212 ms
You are now connected to database "dataload" as user "postgres".
CREATE TABLE
Time: 6.517 ms
INSERT 0 1000000
Time: 28668.343 ms
-- SECOND RUN
dataload=# \i generate_data.sql
Timing is on.
You are now connected to database "postgres" as user "postgres".
DROP DATABASE
Time: 774.061 ms
CREATE DATABASE
Time: 2721.169 ms
You are now connected to database "dataload" as user "postgres".
CREATE TABLE
Time: 7.374 ms
INSERT 0 1000000
Time: 32168.043 ms
dataload=# 

For the instance on the compressed file system:

-- FIRST RUN
postgres=# \i generate_data.sql
Timing is on.
You are now connected to database "postgres" as user "postgres".
psql:generate_data.sql:3: NOTICE:  database "dataload" does not exist, skipping
DROP DATABASE
Time: 0.850 ms
CREATE DATABASE
Time: 4281.965 ms
You are now connected to database "dataload" as user "postgres".
CREATE TABLE
Time: 5.120 ms
INSERT 0 1000000
Time: 30606.966 ms
-- SECOND RUN
dataload=# \i generate_data.sql
Timing is on.
You are now connected to database "postgres" as user "postgres".
DROP DATABASE
Time: 2359.120 ms
CREATE DATABASE
Time: 3267.151 ms
You are now connected to database "dataload" as user "postgres".
CREATE TABLE
Time: 8.665 ms
INSERT 0 1000000
Time: 23474.290 ms
dataload=# 

Despite that the numbers are quite bad (5 seconds to create an empty table) the fastest load was the second one on the compressed file system. So at least it is not slower. I have to admit that I did not do any tuning on the file systems and my VM does not have much memory (512m) which is far too less if you work with ZFS (ZFS needs much memory, at least 1gb).

So, what about the size of the data. First lets check what PostgreSQL is telling us:

-- instance on the uncompressed file system
dataload=# select * from pg_size_pretty ( pg_relation_size ( 'dataload' ));
 pg_size_pretty 
----------------
 81 MB
(1 row)
-- instance on the compressed file system
dataload=# select * from pg_size_pretty ( pg_relation_size ( 'dataload' ));
 pg_size_pretty 
----------------
 81 MB
(1 row)

Exactly the same, which is not surprising as PostgreSQL sees the files as if they would be uncompressed (please be aware that the my_app_table from the last post is still there which is why the file system usage in total is larger than you might expect). It is quite funny on how the size is reported on the compressed file system depending on how you ask.

You can use oid2name to map the file name to a table name:

postgres@centos7:/pgpool/pgdatacompressed/base/24580/ [PG1] oid2name -d dataload -p 5433 -f 24581
From database "dataload":
  Filenode  Table Name
----------------------
     24581    dataload

File 24581 is the table we generated. When you ask for the size by using “du” you get:

postgres@centos7:/pgpool/pgdatacompressed/base/24580/ [PG1] du -h 24581
48M	24581

This is the compressed size. When you use “ls” you get the uncompressed size:

postgres@centos7:/pgpool/pgdatacompressed/base/24580/ [PG1] ls -lh 24581
-rw-------. 1 postgres postgres 81M Sep 30 10:43 24581

What does “df” tell us:

postgres@centos7:/home/postgres/ [PG1] df -h | grep pgdata
pgpool/pgdata            9.5G  437M  9.1G   5% /pgpool/pgdata
pgpool/pgdatacompressed  9.2G  165M  9.1G   2% /pgpool/pgdatacompressed

Not bad, 437M of uncompressed data which is 165M compressed. So, if you are short on space this really can be an option.

Cet article Running PostgreSQL on ZFS on Linux – Compression est apparu en premier sur Blog dbi services.

In a previous post I described the new V$SQL views about rolling invalidation. I did the example with dbms_stats which is able to do rolling invalidation since 11g. But more is coming with 12.2 as you can use rolling invalidation for some DDL.

When you do some DDL on an object, dependent cursors are invalidated. On a busy database it’s something to avoid. I’ve seen recently a peak of hanging sessions during 10 minutes and the root cause was an ALTER TABLE ADD COMMENT. This do not change anything in the execution plan, but the cursors are invalidated. And doing that on a central table can trigger a hard parse storm.

For the example, I’ll reduce the rolling invalidation window to 25 seconds as I don’t want to wait hours:
SQL> alter system set "_optimizer_invalidation_period"=25;
System altered.
Currently 12.2 is available only on Oracle Database Express Cloud Service, but please don’t ask me how I was able to set an underscore parameter there. However you can reproduce the same by waiting 5 hours instead of 25 seconds.

I create a simple table and prepare my query to check cursor information:

SQL> create table DEMO as select * from dual;
Table created.
 
SQL> select sql_text,invalidations,loads,parse_calls,executions,first_load_time,last_load_time,last_active_time,is_rolling_invalid from v$sql where sql_text like 'S%DEMO%';
no rows selected
 

I run a simple query and check the cursor:

SQL> SELECT * FROM DEMO;
 
D
-
X
 
SQL>
SQL> select sql_text,invalidations,loads,parse_calls,executions,first_load_time,last_load_time,last_active_time,is_rolling_invalid from v$sql where sql_text like 'S%DEMO%';
 
SQL_TEXT INVALIDATIONS LOADS PARSE_CALLS EXECUTIONS FIRST_LOAD_TIME LAST_LOAD_TIME LAST_ACTIVE IS_ROLLING_INVALID
------------------- ------------- ---------- ----------- ---------- ------------------- ------------------- ----------- --------------------
SELECT * FROM DEMO 0 1 1 1 2016-09-17/01:04:16 2016-09-17/01:04:16 01:04:16 N

1 parse call, cursor loaded, hard parsed and executed.

I create an index on the table with the DEFERRED INVALIDATION new syntax:


SQL> create index DEMO on DEMO(dummy) deferred invalidation;
Index created.

I see that it is flagged as rolling invalid:

SQL> select sql_text,invalidations,loads,parse_calls,executions,first_load_time,last_load_time,last_active_time,is_rolling_invalid from v$sql where sql_text like 'S%DEMO%';
 
SQL_TEXT INVALIDATIONS LOADS PARSE_CALLS EXECUTIONS FIRST_LOAD_TIME LAST_LOAD_TIME LAST_ACTIVE IS_ROLLING_INVALID
------------------- ------------- ---------- ----------- ---------- ------------------- ------------------- ----------- --------------------
SELECT * FROM DEMO 0 1 1 1 2016-09-17/01:04:16 2016-09-17/01:04:16 01:04:16 Y

When I run it, a timestamp is set within the rolling invalidation window (5 hours by default, but here 25 seconds):

SQL> SELECT * FROM DEMO;
 
D
-
X
 
SQL> select sql_text,invalidations,loads,parse_calls,executions,first_load_time,last_load_time,last_active_time,is_rolling_invalid from v$sql where sql_text like 'S%DEMO%';
 
SQL_TEXT INVALIDATIONS LOADS PARSE_CALLS EXECUTIONS FIRST_LOAD_TIME LAST_LOAD_TIME LAST_ACTIVE IS_ROLLING_INVALID
------------------- ------------- ---------- ----------- ---------- ------------------- ------------------- ----------- --------------------
SELECT * FROM DEMO 0 1 2 2 2016-09-17/01:04:16 2016-09-17/01:04:16 01:04:16 X

As you see, this Y/N flag has a third value to show that it has been executed after being rolling invalidated.

I wait 30 seconds:

SQL> host sleep 30

From that point, the invalidation timestamp has been reached so a new execution will create a new child cursor:

SQL> select sql_text,invalidations,loads,parse_calls,executions,first_load_time,last_load_time,last_active_time,is_rolling_invalid from v$sql where sql_text like 'S%DEMO%';
 
SQL_TEXT INVALIDATIONS LOADS PARSE_CALLS EXECUTIONS FIRST_LOAD_TIME LAST_LOAD_TIME LAST_ACTIVE IS_ROLLING_INVALID
------------------- ------------- ---------- ----------- ---------- ------------------- ------------------- ----------- --------------------
SELECT * FROM DEMO 0 1 2 2 2016-09-17/01:04:16 2016-09-17/01:04:16 01:04:16 X
 
SQL> SELECT * FROM DEMO;
 
D
-
X
 
SQL> select sql_text,invalidations,loads,parse_calls,executions,first_load_time,last_load_time,last_active_time,is_rolling_invalid from v$sql where sql_text like 'S%DEMO%';
 
SQL_TEXT INVALIDATIONS LOADS PARSE_CALLS EXECUTIONS FIRST_LOAD_TIME LAST_LOAD_TIME LAST_ACTIVE IS_ROLLING_INVALID
------------------- ------------- ---------- ----------- ---------- ------------------- ------------------- ----------- --------------------
SELECT * FROM DEMO 0 1 2 2 2016-09-17/01:04:16 2016-09-17/01:04:16 01:04:16 X
SELECT * FROM DEMO 0 1 1 1 2016-09-17/01:04:16 2016-09-17/01:04:45 01:04:45 N


You must be careful here. If you used to check the INVALIDATIONS column, then you may miss the rolling ones. INVALIDATIONS is for parent cursors. IS_ROLLING_INVALID is for invalidated child cursors.

Note that, of course, until the invalidation, the new index will not be used by those cursors. So if you create the index to solve an performance issue you may prefer to invalidate the cursors.

Same can be done with index rebuild:


SQL> alter index DEMO rebuild deferred invalidation;
Index altered.
 
SQL> select sql_text,invalidations,loads,parse_calls,executions,first_load_time,last_load_time,last_active_time,is_rolling_invalid from v$sql where sql_text like 'S%DEMO%';
 
SQL_TEXT INVALIDATIONS LOADS PARSE_CALLS EXECUTIONS FIRST_LOAD_TIME LAST_LOAD_TIME LAST_ACTIVE IS_ROLLING_INVALID
------------------- ------------- ---------- ----------- ---------- ------------------- ------------------- ----------- --------------------
SELECT * FROM DEMO 0 1 2 2 2016-09-17/01:04:16 2016-09-17/01:04:16 01:04:16 X
SELECT * FROM DEMO 0 1 1 1 2016-09-17/01:04:16 2016-09-17/01:04:45 01:04:45 Y
 
SQL> SELECT * FROM DEMO;
 
D
-
X
 
SQL> select sql_text,invalidations,loads,parse_calls,executions,first_load_time,last_load_time,last_active_time,is_rolling_invalid from v$sql where sql_text like 'S%DEMO%';
 
SQL_TEXT INVALIDATIONS LOADS PARSE_CALLS EXECUTIONS FIRST_LOAD_TIME LAST_LOAD_TIME LAST_ACTIVE IS_ROLLING_INVALID
------------------- ------------- ---------- ----------- ---------- ------------------- ------------------- ----------- --------------------
SELECT * FROM DEMO 0 1 2 2 2016-09-17/01:04:16 2016-09-17/01:04:16 01:04:16 X
SELECT * FROM DEMO 0 1 2 2 2016-09-17/01:04:16 2016-09-17/01:04:45 01:04:46 X


Of course, rolling invalidation can happen only for the cursors that do not use the index.

With the same restriction, you can do it when you set an index unusable


SQL> alter index DEMO unusable deferred invalidation;
 
Index altered.
 
SQL> select sql_text,invalidations,loads,parse_calls,executions,first_load_time,last_load_time,last_active_time,is_rolling_invalid from v$sql where sql_text like 'S%DEMO%';
 
SQL_TEXT INVALIDATIONS LOADS PARSE_CALLS EXECUTIONS FIRST_LOAD_TIME LAST_LOAD_TIME LAST_ACTIVE IS_ROLLING_INVALID
------------------- ------------- ---------- ----------- ---------- ------------------- ------------------- ----------- --------------------
SELECT * FROM DEMO 0 1 2 2 2016-09-17/01:04:16 2016-09-17/01:04:16 01:04:16 X
SELECT * FROM DEMO 0 1 2 2 2016-09-17/01:04:16 2016-09-17/01:04:45 01:04:46 X
 
SQL> SELECT * FROM DEMO;
 
D
-
X
 
SQL> select sql_text,invalidations,loads,parse_calls,executions,first_load_time,last_load_time,last_active_time,is_rolling_invalid from v$sql where sql_text like 'S%DEMO%';
 
SQL_TEXT INVALIDATIONS LOADS PARSE_CALLS EXECUTIONS FIRST_LOAD_TIME LAST_LOAD_TIME LAST_ACTIVE IS_ROLLING_INVALID
------------------- ------------- ---------- ----------- ---------- ------------------- ------------------- ----------- --------------------
SELECT * FROM DEMO 0 1 2 2 2016-09-17/01:04:16 2016-09-17/01:04:16 01:04:16 X
SELECT * FROM DEMO 0 1 3 3 2016-09-17/01:04:16 2016-09-17/01:04:45 01:04:46 X


You get the same behavior if you drop the index: cursors that do not use it are no invalidated immediately.

From my tests, you can add DEFERRED INVALIDATION when you MOVE TABLE, but invalidation is immediate. Only when moving partitions, the rolling invalidation occurs.

An alternative to specify DEFERRED INVALIDATION in the DDL is to set it as the default:


SQL> show parameter cursor_invalidation
 
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
cursor_invalidation string IMMEDIATE
 
SQL> alter session set cursor_invalidation=deferred;
 
Session altered.
 


But remember, not all DDL will do rolling invalidation, even when the syntax is accepted.

Cet article Oracle 12cR2: DDL deferred invalidation est apparu en premier sur Blog dbi services.

Oracle incrementally updating backups are used quite often because they are easy to setup and restoring a datafile is very fast. It is very fast because you are not really restoring a datafile, you are switching to the copy, in case something happens. But how do I switch back to the original destination with minimal downtime and with minimal impact on the system?

A quite common scenario is that we have 3 diskgroups, +DATA, +FRA and +REDO with different performance characteristics, like the following:

• +DATA Diskgoup is on fast storage (10k rpm)
• +FRA Diskgroup is on medium storage (7200 rpm)
• +REDO Diskgroup is on very fast storage (15k rpm)

In case we loose now a bigfile with 8TB on the +DATA diskgroup, what options do we  have to restore it. Ok. Lets ask the Oracle Data Recovery Advisor first.  Oracle came up with the following script.

oracle@oel001:/home/oracle/rman/ [OCM121] cat /u00/app/oracle/diag/rdbms/ocm121/OCM121/hm/reco_3905236091.hm
   # restore and recover datafile
   restore ( datafile 9 );
   recover datafile 9;
   sql 'alter database datafile 9 online';

The script does his job, no question about it, but it means, that Oracle would copy 8TB from the +FRA to +DATA and afterwards maybe applying an inc1 and some archivelogs. If we do run this script, we wait for 4h. (suppose that we are copying with 600MB per second, which is very good)

In case your Database has a Standby in a DataGuard configuration, Oracle comes up with the following suggestion.

oracle@oel001:/home/oracle/rman/ [OCM121] cat /u00/app/oracle/diag/rdbms/ocm121/OCM121/hm/reco_4177443733.hm
   # restore from standby and recover datafile
   restore ( datafile 9 from service "sty121" );
   recover datafile 9;
   sql 'alter database datafile 9 online';

Again, it will work, but now Oracle tries to get the 8TB datafile over the network from the standby, which makes it even slower. Unfortunately, the “switch datafile to copy” was not build in, into the Recovery Advisor.

Ok. Lets do it manually and switch to the datafile copy. That takes only a few minutes, and this is the reason why we have incrementally updating backups. To make it as fast as possible.

RMAN> switch datafile 9 to copy;

datafile 9 switched to datafile copy "+FRA/OCM121/DATAFILE/monster.346.919594959"

RMAN> recover datafile 9;

RMAN> sql 'alter database datafile 9 online';

Now we restored and recoverd the 8TB datafile, and users can start working again on that bigfile tablespace. But due to the fact that the +FRA has only medium storage, your application might run slower than before.

Be careful, another issue might pop up after you are already on your datafile copy in the +FRA. If now a backup kicks it (scheduled by cron or something else), then Oracle has to create another 8TB in the +FRA as a new base for his incrementally updating backups, which makes your application even slower and even worse, you might run out of space.

An easy restore might end up now in a quite complex scenario. So what do we do now. First of all, we have to make sure that backups are not scheduled during our restore/recovery, and then we can manually create a new datafile copy in +DATA (of course, after the situation was corrected which lead to the datafile loss). In case you are running 12c, you can use the new feature “Multisection Backup for Datafile Copies”.

RMAN> backup section size 1T as copy datafile 9 format ='+DATA' tag clonefile9;

Now, a small downtime kicks in, when we have to take the datafile offline, switch to our new one in +DATA, recover it, and take it online again.

RMAN> sql 'alter database datafile 9 offline';

RMAN> switch datafile 9 to copy;

RMAN> recover datafile 9;

RMAN> sql 'alter database datafile 9 online';

Uffff … we are ready now, and users can work with the application again which is on the fast storage in +DATA. But wait a second, if we start our RMAN backup again, then Oracle does not regognize  the datafile copy in +FRA anymore as a valid copy for incrementally updataing backups. So, Oracle has to create another 8TB in the +FRA.

Now comes the 1Million $ question. How can we avoid this? We need to tag the datafile copy in +FRA as a valid starting point for incrementally updating backups.

RMAN> catalog datafilecopy '+fra/ocm121/datafile/MONSTER.346.919594959' level 0 TAG 'incr_update';

cataloged datafile copy
datafile copy file name=+FRA/ocm121/datafile/monster.346.919594959 RECID=33 STAMP=919603336

Oracle has the very useful command “catalog” for situations like this. Take care, that you specify “level 0″ and the correct “tag”, otherwise the datafile copy will not be regognized.

Now we are really ready, and we can start the RMAN incremetally backups again, like we did beforehand.

To summarize it:

• Take care of your backups during the restore, it might makes the situation even worse.
• Make use of the new feature “Multisection Backup for Datafile Copies”. It can speed up the creation of your datafile copies quite heavily.
• Use the “catalog” command to tag your datafile copy correctly. It avoids the creation of another 8TB.

Cheers,

William

Cet article Oracle 12c and RMAN switch datafile to copy, is it really so easy? est apparu en premier sur Blog dbi services.

Apache Cassandra is one of the most popular NoSQL databases http://db-engines.com/en/ranking. It is used by many big companies as Facebook, Netflix and many others. Initially the project was initiated by Facebook. In combination with Google BigTable and Amazon DynamoDB, they developed Cassandra. Actually, Cassandra is developed and maintained by the Apache foundation for the community version and by DataStax for the enterprise and commercial versions.

Architecture

Cassandra has a master-less architecture. A cluster is represented as a ring. It’s a peer-to-peer cluster, with no single point of failure.

The Cassandra key features are:

• Distributed
• Decentralized
• Replicated
• Scalable
• Fault-tolerant
• Tunable consistency level
• No Single Point of Failure
• High Available
• Multi data center

Any server node can accept write or read queries from clients. Every server is equal.

Cassandra uses the Gossip protocol for internode communication within the cluster. The Gossip protocol allows the exchanges of information between cluster nodes, such as the status. Each cluster node sends its status every second to three other nodes in the ring. A gossip message contain a version and in case of a conflict the older version is overwritten.

A Partitioned Row Store

Because C* is a NoSQL database, and in the “NoSQL world” no strict standards are defined, there is a lot of misunderstanding around the Cassandra data model. Currently, C* is defined as a column-store oriented DBMS, but there is a big confusion on this definition.

To define, the Cassandra data model you have to decompose it. The picture below will help you to understand the data objects composition.

The Cassandra data model is organized into rows of multiple columns/values pairs. Each row is uniquely identifiable by a key, the Row Key. Rows group columns and super columns. A column is the most basic unit of the data model. All columns are sorted by column key name, and all row key are sorted by row keys.

Each row key, is stored into tables or column-family and both are encapsulated into Keyspaces. Below is a representation of the complete Cassandra data model.

Finally, Apache C* can be defined as a Partitioned row store database.

How data is distributed

Cassandra uses a partitioner (internal component) to distribute data across cluster nodes. A partitioner determines where each piece of data has to be stored. The partitioning process is completely automatic and transparent.

Basically the partitioner is a hash function, computes a token for each partition key (Partition Key = Row Key). Depending on the partitioning strategy, each node of the cluster is responsible of a token. Below a picture for the partitioning process.

Three partitioning strategies are available:

• Murmur3Partitioner
• RandomPartitioner
• ByteOrderedPartitioner

Replication

Cassandra can store multiple copies of data on multiple nodes in order to ensure high availability and fault tolerance. Two important concepts are used for C* replication.

Replication strategy: determines in which node data is placed

Replication factor: determines the number of nodes where data is placed. A replication factor of 1 means that the data is replicated one time in one node.

The client connects to any node in the ring. This node became the coordinator, then the coordinator determines where the data must be stored. Finally, the data is replicated from one node to another in the sense of a clock hand.

A quick overview of Cassandra main concepts has been done in this post in order to understand more precisely how Apache Cassandra works. Of course Cassandra concepts are more complex and will be described in future blog posts.

Cet article Apache Cassandra overview est apparu en premier sur Blog dbi services.

As with any other software there comes the time when you need to patch your Postgres Plus Advanced Server instances. Is that different from patching Community PostgreSQL? Yes and no The difference is that you need a subscription to get access to the EDB Customer Portal for being able to download the patch. This is pretty much the same as with My Oracle Support where you need a customer support identifier mapped to your account for being able to download patches, to have access to the knowledge base and for being able to open cases in case you run into troubles which you are not able to solve yourself.

Assuming you have access to the EDB customer portal and you downloaded the patch for your base release the procedure is pretty simple. For this little demo I am running the 9.5.0.5 base release of Postgres Plus Advanced Server:

postgres@centos7:/home/postgres/ [PG2] psql
psql.bin (9.5.0.5)
Type "help" for help.

postgres=# select version();
                                                   version                                                    
--------------------------------------------------------------------------------------------------------------
 EnterpriseDB 9.5.0.5 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 4.1.2 20080704 (Red Hat 4.1.2-55), 64-bit
(1 row)

postgres=# 

This is the release you can download from the EDB website for testing. The patch I downloaded from the portal is this one:

postgres@centos7:/home/postgres/ [PG2] cd /u01/app/postgres/software/
postgres@centos7:/u01/app/postgres/software/ [PG2] ls -la
total 27216
drwxrwxr-x. 2 postgres postgres       51 Oct  5 10:16 .
drwxrwxr-x. 5 postgres postgres       47 Jun 15 13:10 ..
-rw-rw-r--. 1 postgres postgres 27868299 Oct  5 10:16 postgresplusas-9.5.4.9-1-linux-x64.run
postgres@centos7:/u01/app/postgres/software/ [PG2] chmod +x postgresplusas-9.5.4.9-1-linux-x64.run 

This should patch my base release to currently latest release which is 9.5.4.9-1. How does it work? Lets execute the binary and see what happens:

postgres@centos7:/u01/app/postgres/software/ [PG2] ./postgresplusas-9.5.4.9-1-linux-x64.run 
Language Selection

Please select the installation language
[1] English - English
[2] Japanese - 日本語
[3] Simplified Chinese - 简体中文
[4] Traditional Chinese - 繁体中文
[5] Korean - 한국어
Please choose an option [1] : 1

Error: There has been an error.
This installer requires root privileges. Please become superuser before 
executing the installer
Press [Enter] to continue:

A no go for most of the cases. Running installers as root is not a good practice and should be avoided whenever possible. But, luckily, as with the base release installer the patch itself can be installed in “extract only” mode:

postgres@centos7:/u01/app/postgres/software/ [PG2] ./postgresplusas-9.5.4.9-1-linux-x64.run --extract-only yes --prefix /u01/app/postgres/product/95edb/db_5/9.5AS/
Language Selection

Please select the installation language
[1] English - English
[2] Japanese - 日本語
[3] Simplified Chinese - 简体中文
[4] Traditional Chinese - 繁体中文
[5] Korean - 한국어
Please choose an option [1] : 1
----------------------------------------------------------------------------
Welcome to the Postgres Plus Advanced Server Setup Wizard.

----------------------------------------------------------------------------
Please specify the directory where Postgres Plus Advanced Server will be 
installed.

Installation Directory [/u01/app/postgres/product/95edb/db_5/9.5AS]: 

----------------------------------------------------------------------------
Setup is now ready to begin installing Postgres Plus Advanced Server on your 
computer.

Do you want to continue? [Y/n]: y

----------------------------------------------------------------------------
Please wait while Setup installs Postgres Plus Advanced Server on your computer.

 Installing Database Server
 0% ______________ 50% ______________ 100%
 #########################################

----------------------------------------------------------------------------
Setup has finished installing Postgres Plus Advanced Server on your computer.

postgres@centos7:/u01/app/postgres/software/ [PG2] 

Looks good, but you never should do this when your PostgreSQL is running, otherwise you’ll get this:

2016-10-05 10:22:43 CEST LOG:  server process (PID 4359) was terminated by signal 11: Segmentation fault
2016-10-05 10:22:43 CEST LOG:  terminating any other active server processes
2016-10-05 10:22:43 CEST WARNING:  terminating connection because of crash of another server process
2016-10-05 10:22:43 CEST DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2016-10-05 10:22:43 CEST HINT:  In a moment you should be able to reconnect to the database and repeat your command.
2016-10-05 10:22:43 CEST LOG:  statistics collector process (PID 3324) was terminated by signal 11: Segmentation fault
2016-10-05 10:22:43 CEST LOG:  all server processes terminated; reinitializing

Always shutdown, before you begin to patch. In my case I just started the instance again and I am on the current release:

postgres@centos7:/u01/app/postgres/software/ [PG2] sqh
psql.bin (9.5.4.9)
Type "help" for help.

postgres=# select version();
                                                   version                                                    
--------------------------------------------------------------------------------------------------------------
 EnterpriseDB 9.5.4.9 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 4.1.2 20080704 (Red Hat 4.1.2-55), 64-bit
(1 row)

postgres=# 

Simple and fast. If you prepare this very well your downtime will be around one minute. I can already hear the question: Can I switchover to my standby, apply the patch on the master, switchback and then proceed on the standby for reducing the downtime even more? This will be a topic for another post.

PS: Of course you can also prepare a brand new home for the patched binaries and then shutdown your instance, switch to the new binaries and start again from there.

PS2: Just in case you are not aware of: As we have established a partnership with EnterpriseDB in the past you can obtain (an) EDB subscription(s) easily from us. Of course we would do a review of what you really need before. It is not always required to go for the Postgres Plus version. Community PostgreSQL works very well in the most cases and can be backed by a EDB subscription as well, if required.

Cet article How to patch Postgres Plus Advanced Server est apparu en premier sur Blog dbi services.

This blog post is about the configuration of Apache Cassandra for running a cluster of 3 instances on a single host. Basically C* is not really made to run it in a multi instances environment, but for many cases and reasons you might run a C* cluster on a single server. The principal reason is for tests cases.

Context

The objective is to have a Cassandra ring of 3 instances on 1 host. For that, we will use:

• apache-cassandra-3.0.9 (community version)
• Separated disks for a segregation between binaries, data and logs: binaries, data and logs will be in separated mount points
• Multiple IPs interfaces for each instance (virtual interfaces in our case)
• Instances name: csd1, csd2, csd3
• Cluster name: dbitest

Binary installation

Download the binaries from the Apache website, http://www.apache.org/dyn/closer.lua/cassandra/3.0.9/apache-cassandra-3.0.9-bin.tar.gz and install it. In this post we are going to install the 3.0.9 release.

wget http://www.pirbot.com/mirrors/apache/cassandra/3.0.9/apache-cassandra-3.0.9-bin.tar.gz

Extract it:

tar -zxvf apache-cassandra-3.0.9-bin.tar.gz

Place the software into the product directory:

cp –r apache-cassandra-3.0.9 /u00/app/cassandra/product/

Instances configuration

First, we will create the admin directory for each instance. The admin directory contains the following sub folders:

• /u00/app/cassandra/admin/csd$i
  • /backup: soft link to the backup mount point
  • /dump: soft link to the dump mount point
  • /etc: Cassandra configuration files, such as Cassandra.yaml, logback.xml and Cassandra-env.sh.
  • /log: soft link to the log file from the log mount point
  • /pid: contain the pid of the C* process

To create these directories, use the following loop.

for i in 1 2 3 ;
do mkdir -p /u00/app/cassandra/admin/csd$i &&
mkdir /u00/app/cassandra/admin/csd$i/backup &&
mkdir /u00/app/cassandra/admin/csd$i/dump &&
mkdir /u00/app/cassandra/admin/csd$i/etc &&
mkdir /u00/app/cassandra/admin/csd$i/log &&
mkdir /u00/app/cassandra/admin/csd$i/pid &&
mkdir /u00/app/cassandra/admin/csd$i/product;
done

Next copy the C* configuration files into each admin/csd$i/etc directory. Every instance has its own configuration files.

for i in 1 2 3;
do cp -r /u00/app/cassandra/admin/csd$i/product/apache-cassandra-3.0.9/conf/* /u00/app/cassandra/admin/csd$i/etc/;
done

Now, create the three data directories for each instance.

for i in 1 2 3;
do
sudo mkdir -p /u01/cassandradata/csd$i &&
sudo chown -R cassandra:cassandra /u01/cassandradata/csd$i;
done

Configure virtual network interfaces

You must create additional virtual network interfaces for each C* instance, to use a different listen_address and rpc_address for each instance.

sudo ifconfig enp0s3:0 192.168.56.105
sudo ifconfig enp0s3:1 192.168.56.106

Then, you have 3 interfaces (default + 2 virtuals), for your C* instances.

enp0s3: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.56.104  …

enp0s3:0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.56.105  …

enp0s3:1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.56.106  …

enp0s8: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        …

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
…

As all the required files are in place, we can now change the 3 main configuration files, to set up the three instances (csd1, csd2 and csd3).

cassandra-env.sh:

Use a different “JMX_PORT” for each node. For instance: 7199 for csd1, 7299 for csd2 and 7399 for csd3.

JMX will bind to the local host IP by default, you can use the default.

cassandra.yaml

The central configuration file of Apache Cassandra. Change the following parameters:

• cluster_name: ‘dbitest’ All instances must have the same cluster name
• commitlog_directory: /u01/cassandradata/csd$i/commitlog
• data_file_directories:/u01/cassandradata/csd$i/data
• saved_caches_directory: /u01/cassandradata/csd$i/saved_caches
• listen_address: 192.168.56.104
• rpc_address: 192.168.56.104
• seeds: specify the list of IP of the seed nodes

logback.xml

The configuration file for C* logs. Make the following changes:

<file>${cassandra.logdir}/system.log</file> to <file>/u00/app/cassandra/admin/csd$i/log/system.log</file>

<fileNamePattern>${cassandra.logdir}/system.log.%i.zip</fileNamePattern> to 
<fileNamePattern>/u00/app/cassandra/admin/csd$i/log/system.log.%i.zip</fileNamePattern>

<file>${cassandra.logdir}/debug.log</file> to <file>/u00/app/cassandra/admin/csd$i/debug.log</file>

<fileNamePattern>${cassandra.logdir}/debug.log.%i.zip</fileNamePattern> to
<fileNamePattern>/u00/app/cassandra/admin/csd1/log/debug.log.%i.zip</fileNamePattern>

Make the change for the three instances.

Starting the nodes

Before starting each node, you must dynamically set the environment variables for each of them.

$CASSANDRA_HOME=/u00/app/cassandra/product/apache-cassandra-3.0.9
$CASSANDRA_CONF=/u00/app/cassandra/admin/csd$i/etc/

After, setting the variables for each node you can start Apache Cassandra by command-line:

./$CASSANDRA_HOME/bin/cassandra –f

Execute the same command in a separated shell, for each node. Do not forget to set the environment variables $CASSANDRA_HOME and $CASSANDRA_CONF.

Cluster status

Verify the cluster status with nodetool utility. Check if all nodes are up and running.

[cassandra@test bin]$ ./nodetool status
 Datacenter: datacenter1
 =======================
 Status=Up/Down|/ State=Normal/Leaving/Joining/Moving
 --  Address         Load       Tokens       Owns (effective)  Host ID                               Rack
 UN  192.168.56.104  263.42 KB  256          64.5%             cddee7f4-c0d5-4cba-9ddc-b773a08a9245  rack1
 UN  192.168.56.105  212.19 KB  256          67.5%             19cd17d2-1aeb-48e5-9299-7a0282c2e92e  rack1
 UN  192.168.56.106  147.54 KB  256          68.0%             f65d5b92-ed3b-4f68-b93d-72e6162eafca  rack1

Enjoy

Cet article Running a Cassandra cluster in a single server est apparu en premier sur Blog dbi services.

For quite a long time I did not work anymore with PL/SQL and was quite happy when I had the chance to review some code at a customer. The status today: I am not that happy anymore Let me explain why and show you some examples on what was discovered. Of course all of the identifiers have been obfuscated and this is not to blame anyone. It is more to make people aware of what you should not do and that you have to be as exact as possible when you do programming. Even more important: Code that you write must be maintainable and understandable by others.

We start with a simple “if-then-else-end if” block:

         IF c <= d
         THEN
            IF c = d
            THEN
               l2 := l2 || l3;
            ELSE
               l2 := l2 || l3 || ',';
            END IF;
         END IF;

So what is wrong with this? The code does what it is supposed to do, true. But this is more complicated than it needs to be. Lets look at the first line:

         IF c <= d

If we enter this “IF” then we know that c is either less than d or equal to d, correct? On line 3 we know that c is d if we enter that “IF”. What does this imply for line 6 (the “ELSE”)?
It implies that c is less than d when we enter the “ELSE”. But then we could also write it like this:

            IF c = d
            THEN
               l2 := l2 || l3;
            ELSIF c < d THEN
               l2 := l2 || l3 || ',';
            END IF;

This is much more clear and less code. We are only interested if c is equal to d or less than d, that’s it. Then we should design the code exactly for that use case.

The next example is about the usage of a record. This is the code block:

         ...

         ll_col1 SCHEMA.TABLE.COLUMN1%TYPE;
         ll_col2 SCHEMA.TABLE.COLUMN2%TYPE;
         ll_col3 SCHEMA.TABLE.COLUMN3%TYPE;
         ll_col4 SCHEMA.TABLE.COLUMN4%TYPE;
         ...
         DECLARE
            TYPE MyRecord IS RECORD
            (
               l_col1    SCHEMA.TABLE.COLUMN1%TYPE,
               l_col2    SCHEMA.TABLE.COLUMN2%TYPE,
               l_col3    SCHEMA.TABLE.COLUMN3%TYPE,
               l_col4    SCHEMA.TABLE.COLUMN4%TYPE
            );
            rec1   MyRecord;
         BEGIN
            v_sql := 'SELECT col1, col2, col3, col4 FROM SCHEMA.TABLE WHERE col3 = '''
               || ll_col3
               || '''';
 
            EXECUTE IMMEDIATE v_sql INTO rec1;

            ll_col1 := rec1.l_col1;
            ll_col2 := rec1.l_col2;
            ll_col3 := rec1.l_col3;
            ll_col4 := rec1.l_col4;
         EXCEPTION
            WHEN NO_DATA_FOUND
            THEN
               ll_col3 := 0;
         END;

Beside that declaring a new block inside an existing block is really ugly what is wrong here? There is the definition of a new record which is then used to fetch the data from the dynamic sql statement. Nothing wrong here. But then the fields of this record are written back to four variables which are valid in and outside of that block, why that? Probably because the record is only valid in the “declare-begin-end” block. But for what do I need the record then at all? Really confusing

The next one is more about how easy it is to understand code written by someone else. This is the code block:

      SELECT COUNT (*)
        INTO l_count
        FROM USER_INDEXES
       WHERE INDEX_NAME = '' || l_index_name || '';

      IF l_count > 0
      THEN
         EXECUTE IMMEDIATE 'DROP INDEX ' || '' || l_index_name || '';
      END IF;

I really had to read this carefully until I understood why it was written this way. Probably the developer had this intension: “I want to know if a specific index exists and if yes, then I want to drop it”.
But what he did actually code is: I want to know if there are multiple indexes with the same name. As the query is against user_tables the answer can not be more than 1, correct? But then the correct and much more easy to understand way of writing this would be:

      SELECT COUNT (*)
        INTO l_count
        FROM USER_INDEXES
       WHERE INDEX_NAME = '' || l_index_name || '';

      IF l_count = 1
      THEN
         EXECUTE IMMEDIATE 'DROP INDEX ' || '' || l_index_name || '';
      END IF;

This is just a small change and does not affect the functionality at all but it is much more clear. Even more clear would be something like this:

      BEGIN
        SELECT 'index_exists'
          INTO l_exists
          FROM USER_INDEXES
         WHERE INDEX_NAME = '' || l_index_name || '';
      EXCEPTION
        WHEN NO_DATA_FOUND THEN l_exists := 'index_does_not_exist';
      END;
      IF l_exists = 'index_exists'
      THEN
         EXECUTE IMMEDIATE 'DROP INDEX ' || '' || l_index_name || '';
      END IF;

Using our language we can make the code much more easy to understand. Beside that the concatenation in the where clause and in the execute immediate statement is not required so the final code could look like this:

      BEGIN
        SELECT 'index_exists'
          INTO l_exists
          FROM USER_INDEXES
         WHERE INDEX_NAME = l_index_name;
      EXCEPTION
        WHEN NO_DATA_FOUND THEN l_exists := 'index_does_not_exist';
      END;
      IF l_exists = 'index_exists'
      THEN
         EXECUTE IMMEDIATE 'DROP INDEX ' || l_index_name;
      END IF;

In the next post we’ll look at some other examples which can be improved by changing the way we think about what we code and how others may interpret it.

Cet article Fun with PL/SQL code reviews – Part 1 est apparu en premier sur Blog dbi services.

Some time ago, I had to deal with a new partitioning scenario that included sliding windows stuff for mainly archiving purpose. Regarding the customer context, I used some management scripts that include this time dropping oldest partition. We didn’t care about data oldest than 2 years. Usually in this case, I use a method that consists in dropping data by switching first the oldest partition to a staging table and then truncate it. Finally we may safely merge oldest partitions and avoid any data movement. At a first glance, it seems to be a complex process for dropping data but until SQL Server 2014 there is no way to do better in order to minimize operation logging.

This week, I had the opportunity to work with SQL Server 2016 to learn about new partition improvements. By the way, the only thing I could find out from my different internet researches concerned the new command TRUNCATE TABLE WITH PARTITIONS.

My first feeling was it is not a very exciting feature in contrast to previous versions that provided a mixture of both performance and maintenance improvements in this field. But after investigating further, I was able to point out some advantages to use this command. Let’s go back to my sliding windows scenario. In order to drop data from my oldest partition I have to:

• Switch the oldest partition to a staging table
• Drop data from this staging table with TRUNCATE command in order to minimize transaction logging
• Execute MERGE command in order to slide all partitions to the left side

What about introducing the new TRUNCATE TABLE command in this scenario?

In fact, it will simplify the above process by replacing step 1 and step 2 by the TRUNCATE command at the partition level. The new scenario becomes:

• TRUNCATE TABLE at the corresponding partition
• Execute MERGE command in order to slide all partitions to the left

The only instruction I need to use is as follows:

TRUNCATE TABLE [dbo].[FactOnlineSales]
WITH ( PARTITIONS (2) );

What about locking?

As expected, SQL Server will use a lock granularity hierarchy with a mixture of Sch-S, Sch-M and X locks regarding the corresponding locked resource. You may see two allocation units in my case because I’m using a partitioned clustered columnstore index in this demo. As a reminder, compressed columnstore segments are stored in LOB.

What about logging?

Well, if I refer to the corresponding records into the transaction log file, TRUNCATE partition command seems to act as a normal TRUNCATE operation. Firstly, we may notice few records generated related to marking the concerned structures to drop and then the deferred drop mechanism comes into play by deallocating them.

…

…

…

Happy partitioning!

Cet article SQL Server 2016: TRUNCATE PARTITIONS with sliding Windows scenarios est apparu en premier sur Blog dbi services.

In the last post we looked at how you can patch a Postgres Plus Advanced server. Wouldn’t it be nice, in a standby configuration, to patch the standby first without touching the master, then do a controlled switchover and finally patch the old master? In case this is a configuration with EDB Failover Manager the only downtime which would happen is the relocation of the VIP from one node to another (if you use a VIP). Without using a VIP but using pgpool-II the downtime is even less. Lets see if it works by starting from my usual EDB Failover Manager configuration.

This is the current status of my failover cluster:

[root@edbbart efm-2.1]# /usr/efm-2.1/bin/efm cluster-status efm
Cluster Status: efm
VIP: 192.168.22.250
Automatic failover is disabled.

	Agent Type  Address              Agent  DB       Info
	--------------------------------------------------------------
	Witness     192.168.22.244       UP     N/A       
	Master      192.168.22.243       UP     UP        
	Standby     192.168.22.245       UP     UP        

Allowed node host list:
	192.168.22.244 192.168.22.245 192.168.22.243

Membership coordinator: 192.168.22.245

Standby priority host list:
	192.168.22.245

Promote Status:

	DB Type     Address              XLog Loc         Info
	--------------------------------------------------------------
	Master      192.168.22.243       0/4A000140       
	Standby     192.168.22.245       0/4A000140       

	Standby database(s) in sync with master. It is safe to promote.

All is fine, I have one master, one standby and one witness. Going straight forward lets shutdown the standby (please notice that I have disabled auto failover):

Shutdown the standby database:

postgres@edbppasstandby:/home/postgres/ [pg950] pg_ctl -D /u02/pgdata/PGSITE2 stop -m fast
waiting for server to shut down.... done
server stopped

What happened to my cluster?

[root@edbbart efm-2.1]# /usr/efm-2.1/bin/efm cluster-status efm
Cluster Status: efm
VIP: 192.168.22.250
Automatic failover is disabled.

	Agent Type  Address              Agent  DB       Info
	--------------------------------------------------------------
	Witness     192.168.22.244       UP     N/A       
	Master      192.168.22.243       UP     UP        
	Standby     192.168.22.245       UP     UP        

Allowed node host list:
	192.168.22.244 192.168.22.245 192.168.22.243

Membership coordinator: 192.168.22.245

Standby priority host list:
	192.168.22.245

Promote Status:

	DB Type     Address              XLog Loc         Info
	--------------------------------------------------------------
	Master      192.168.22.243       0/4A000140       
	Unknown     192.168.22.245       UNKNOWN          Connection to 192.168.22.245:4445 refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.

	One or more standby databases are not in sync with the master database.
[root@edbbart efm-2.1]# 

Not really surprising EFM complains that the standby is not reachable anymore. Thats fine. Lets patch the standby:

postgres@edbppasstandby:/u01/app/postgres/software/ [pg950] chmod +x postgresplusas-9.5.4.9-1-linux-x64.run
postgres@edbppasstandby:/u01/app/postgres/software/ [pg950] ./postgresplusas-9.5.4.9-1-linux-x64.run --extract-only true --prefix /u01/app/postgres/product/95/db_1/9.5AS/
Language Selection

Please select the installation language
[1] English - English
[2] Japanese - 日本語
[3] Simplified Chinese - 简体中文
[4] Traditional Chinese - 繁体中文
[5] Korean - 한국어
Please choose an option [1] : 1
----------------------------------------------------------------------------
Welcome to the Postgres Plus Advanced Server Setup Wizard.

----------------------------------------------------------------------------
Please specify the directory where Postgres Plus Advanced Server will be 
installed.

Installation Directory [/u01/app/postgres/product/95/db_1/9.5AS]: 

----------------------------------------------------------------------------
Setup is now ready to begin installing Postgres Plus Advanced Server on your 
computer.

Do you want to continue? [Y/n]: Y

----------------------------------------------------------------------------
Please wait while Setup installs Postgres Plus Advanced Server on your computer.

 Installing Database Server
 0% ______________ 50% ______________ 100%
 #########################################

----------------------------------------------------------------------------
Setup has finished installing Postgres Plus Advanced Server on your computer.

postgres@edbppasstandby:/u01/app/postgres/software/ [pg950] 

… bring it up again:

postgres@edbppasstandby:/home/postgres/ [pg950] pg_ctl -D /u02/pgdata/PGSITE2 start
server starting

… checking the PostgreSQL log file all is fine, streaming restarted:

2016-10-05 11:35:25.745 GMT - 2 - 4984 -  - @ LOG:  entering standby mode
2016-10-05 11:35:25.751 GMT - 3 - 4984 -  - @ LOG:  consistent recovery state reached at 0/4A000108
2016-10-05 11:35:25.751 GMT - 4 - 4984 -  - @ LOG:  redo starts at 0/4A000108
2016-10-05 11:35:25.751 GMT - 5 - 4984 -  - @ LOG:  invalid record length at 0/4A000140
2016-10-05 11:35:25.751 GMT - 4 - 4982 -  - @ LOG:  database system is ready to accept read only connections
2016-10-05 11:35:25.755 GMT - 1 - 4988 -  - @ LOG:  started streaming WAL from primary at 0/4A000000 on timeline 8

What is the status of EFM?

postgres@edbppasstandby:/home/postgres/ [pg950] efmstat
Cluster Status: efm
VIP: 192.168.22.250
Automatic failover is disabled.

	Agent Type  Address              Agent  DB       Info
	--------------------------------------------------------------
	Witness     192.168.22.244       UP     N/A       
	Master      192.168.22.243       UP     UP        
	Idle        192.168.22.245       UP     UNKNOWN   

Allowed node host list:
	192.168.22.244 192.168.22.245 192.168.22.243

Membership coordinator: 192.168.22.245

Standby priority host list:
	(List is empty.)

Promote Status:

	DB Type     Address              XLog Loc         Info
	--------------------------------------------------------------
	Master      192.168.22.243       0/4A000140       

	No standby databases were found.

Idle Node Status (idle nodes ignored in XLog location comparisons):

	Address              XLog Loc         Info
	--------------------------------------------------------------
	192.168.22.245       0/4A000140       DB is in recovery.

Status “Idle” for the standby which is fine, just resume:

postgres@edbppasstandby:/home/postgres/ [pg950] sudo /usr/efm-2.1/bin/efm resume efm
Resume command successful on local agent.
postgres@edbppasstandby:/home/postgres/ [pg950] efmstat
Cluster Status: efm
VIP: 192.168.22.250
Automatic failover is disabled.

	Agent Type  Address              Agent  DB       Info
	--------------------------------------------------------------
	Witness     192.168.22.244       UP     N/A       
	Standby     192.168.22.245       UP     UP        
	Master      192.168.22.243       UP     UP        

Allowed node host list:
	192.168.22.244 192.168.22.245 192.168.22.243

Membership coordinator: 192.168.22.245

Standby priority host list:
	192.168.22.245

Promote Status:

	DB Type     Address              XLog Loc         Info
	--------------------------------------------------------------
	Master      192.168.22.243       0/4A000140       
	Standby     192.168.22.245       0/4A000140       

	Standby database(s) in sync with master. It is safe to promote.

… and everything is back as it should be. Time to switchover:

postgres@edbppasstandby:/home/postgres/ [PGSITE2] sudo /usr/efm-2.1/bin/efm promote efm -switchover
Promote/switchover command accepted by local agent. Proceeding with promotion and will reconfigure original master. Run the 'cluster-status' command for information about the new cluster state.

The master and the standby should have switched its roles:

postgres@edbppasstandby:/home/postgres/ [PGSITE2] efmstat
Cluster Status: efm
VIP: 192.168.22.250
Automatic failover is disabled.

	Agent Type  Address              Agent  DB       Info
	--------------------------------------------------------------
	Master      192.168.22.245       UP     UP        
	Witness     192.168.22.244       UP     N/A       
	Standby     192.168.22.243       UP     UP        

Allowed node host list:
	192.168.22.244 192.168.22.245 192.168.22.243

Membership coordinator: 192.168.22.245

Standby priority host list:
	192.168.22.243

Promote Status:

	DB Type     Address              XLog Loc         Info
	--------------------------------------------------------------
	Master      192.168.22.245       0/4B0001A8       
	Standby     192.168.22.243       0/4B0001A8       

	Standby database(s) in sync with master. It is safe to promote.
postgres@edbppasstandby:/home/postgres/ [PGSITE2] 

Same procedure again, stop the standby:

postgres@edbppas:/home/postgres/ [PGSITE1] pg_ctl -D /u02/pgdata/PGSITE1 stop -m fast
waiting for server to shut down.... done
server stopped

postgres@edbppasstandby:/home/postgres/ [PGSITE2] efmstat
Cluster Status: efm
VIP: 192.168.22.250
Automatic failover is disabled.

	Agent Type  Address              Agent  DB       Info
	--------------------------------------------------------------
	Standby     192.168.22.243       UP     UP        
	Witness     192.168.22.244       UP     N/A       
	Master      192.168.22.245       UP     UP        

Allowed node host list:
	192.168.22.244 192.168.22.245 192.168.22.243

Membership coordinator: 192.168.22.245

Standby priority host list:
	192.168.22.243

Promote Status:

	DB Type     Address              XLog Loc         Info
	--------------------------------------------------------------
	Master      192.168.22.245       0/4B0001A8       
	Unknown     192.168.22.243       UNKNOWN          Connection to 192.168.22.243:4445 refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.

	One or more standby databases are not in sync with the master database.

Apply the patch:

postgres@edbppas:/u01/app/postgres/software/ [PGSITE1] ./postgresplusas-9.5.4.9-1-linux-x64.run --extract-only true --prefix /u01/app/postgres/product/95/db_1/9.5AS/
Language Selection

Please select the installation language
[1] English - English
[2] Japanese - 日本語
[3] Simplified Chinese - 简体中文
[4] Traditional Chinese - 繁体中文
[5] Korean - 한국어
Please choose an option [1] : 1
----------------------------------------------------------------------------
Welcome to the Postgres Plus Advanced Server Setup Wizard.

----------------------------------------------------------------------------
Please specify the directory where Postgres Plus Advanced Server will be 
installed.

Installation Directory [/u01/app/postgres/product/95/db_1/9.5AS]: 

----------------------------------------------------------------------------
Setup is now ready to begin installing Postgres Plus Advanced Server on your 
computer.

Do you want to continue? [Y/n]: y

----------------------------------------------------------------------------
Please wait while Setup installs Postgres Plus Advanced Server on your computer.

 Installing Database Server
 0% ______________ 50% ______________ 100%
 #########################################

----------------------------------------------------------------------------
Setup has finished installing Postgres Plus Advanced Server on your computer.

Startup again:

postgres@edbppas:/u01/app/postgres/software/ [PGSITE1] pg_ctl -D /u02/pgdata/PGSITE1 start
server starting

Streaming restarted:

2016-10-05 11:45:36.807 GMT - 2 - 4883 -  - @ LOG:  entering standby mode
2016-10-05 11:45:36.810 GMT - 3 - 4883 -  - @ LOG:  consistent recovery state reached at 0/4B0000C8
2016-10-05 11:45:36.810 GMT - 4 - 4883 -  - @ LOG:  redo starts at 0/4B0000C8
2016-10-05 11:45:36.810 GMT - 5 - 4883 -  - @ LOG:  invalid record length at 0/4B0001A8
2016-10-05 11:45:36.810 GMT - 4 - 4881 -  - @ LOG:  database system is ready to accept read only connections
2016-10-05 11:45:36.815 GMT - 1 - 4887 -  - @ LOG:  started streaming WAL from primary at 0/4B000000 on timeline 9

Same status “Idle” as before:

postgres@edbppasstandby:/home/postgres/ [PGSITE2] efmstat
Cluster Status: efm
VIP: 192.168.22.250
Automatic failover is disabled.

	Agent Type  Address              Agent  DB       Info
	--------------------------------------------------------------
	Idle        192.168.22.243       UP     UNKNOWN   
	Witness     192.168.22.244       UP     N/A       
	Master      192.168.22.245       UP     UP        

Allowed node host list:
	192.168.22.244 192.168.22.245 192.168.22.243

Membership coordinator: 192.168.22.245

Standby priority host list:
	(List is empty.)

Promote Status:

	DB Type     Address              XLog Loc         Info
	--------------------------------------------------------------
	Master      192.168.22.245       0/4B0001A8       

	No standby databases were found.

Idle Node Status (idle nodes ignored in XLog location comparisons):

	Address              XLog Loc         Info
	--------------------------------------------------------------
	192.168.22.243       0/4B0001A8       DB is in recovery.

Resume:

postgres@edbppas:/home/postgres/ [PGSITE1] sudo /usr/efm-2.1/bin/efm resume efm
Resume command successful on local agent.

Fully back:

postgres@edbppasstandby:/home/postgres/ [PGSITE2] efmstat
Cluster Status: efm
VIP: 192.168.22.250
Automatic failover is disabled.

	Agent Type  Address              Agent  DB       Info
	--------------------------------------------------------------
	Standby     192.168.22.243       UP     UP        
	Witness     192.168.22.244       UP     N/A       
	Master      192.168.22.245       UP     UP        

Allowed node host list:
	192.168.22.244 192.168.22.245 192.168.22.243

Membership coordinator: 192.168.22.245

Standby priority host list:
	192.168.22.243

Promote Status:

	DB Type     Address              XLog Loc         Info
	--------------------------------------------------------------
	Master      192.168.22.245       0/4B0001A8       
	Standby     192.168.22.243       0/4B0001A8       

	Standby database(s) in sync with master. It is safe to promote.

Works like a charm. The organizational overhead is much more than what you actually need to do. Technically this is a task of a few minutes.

Cet article How to patch Postgres Plus Advanced Server in a Standby configuration est apparu en premier sur Blog dbi services.

Sometimes, if you kick off a huge load or transformation job, your Archive Destination might right full faster than your RMAN backup job cleans up the destination. To avoid an Archiver Stuck in such situations, the “oradebug suspend/resume” can be helpful, or the UNIX kill command. Usually, the “oradebug suspend” and the UNIX kill command work quite well.

Before kicking off you SQL script, get all the information you need from your session.

select s.username as Username, 
       s.machine as Machine, 
	   s.client_info as Client_Info, 
	   s.module as Module, 
	   s.action as Action, 
	   s.sid as SessionID,
	   p.pid as ProcessID,
	   p.spid as "UNIX ProcessID"
from
v$session s, v$process p
where s.sid = sys_context ('userenv','sid')
and s.PADDR = p.ADDR;

USERNAME     MACHINE      CLIENT_INFO  MODULE                           ACTION        SESSIONID  PROCESSID UNIX ProcessID
------------ ------------ ------------ -------------------------------- ------------ ---------- ---------- ------------------------
SYS          oel001                    sqlplus@oel001 (TNS V1-V3)                           148         69 7186

In another SQL session you can now set the PID with setorapid or the Server Process ID with setospid. HINT: When the Oracle multiprocess/multithread feature is enabled, RDBMS processes are mapped to threads running in operating system processes, and the SPID identifier is not unique for RDBMS processes. When the Oracle multiprocess/multithread feature is not enabled on UNIX systems, the SPID identifier is unique for RDBMS processes.

SQL> oradebug setorapid 69
Oracle pid: 69, Unix process pid: 7186, image: oracle@oel001 (TNS V1-V3)
SQL> -- oradebug setospid 7186
SQL> oradebug suspend
Statement processed.

-- Now your session is suspended and any command executed by the suspended session is hanging, even select's
-- SQL> select * from dual;

-- Now you can take your time and clean up the archive destination e.g. by moving all archivelogs
-- to tape and delete those in the archive destination afterwards "RMAN> backup archivelog all delete all input;"
-- After the job is done, resume your operation.

SQL> oradebug resume
Statement processed.

-- Now the "select * from dual" comes back.
SQL> select * from dual;

D
-
X

In case you are running 11.2.0.2, it might happens that you see an ORA-600 after running oradebug suspend. No problem, in those cases we can achive the same thing with the UNIX kill command as well.

On Linux you would run:

$ kill -sigstop $SPID
$ kill -sigcont $SPID

$ kill -l
 1) SIGHUP       2) SIGINT       3) SIGQUIT      4) SIGILL       5) SIGTRAP
 6) SIGABRT      7) SIGBUS       8) SIGFPE       9) SIGKILL     10) SIGUSR1
11) SIGSEGV     12) SIGUSR2     13) SIGPIPE     14) SIGALRM     15) SIGTERM
16) SIGSTKFLT   17) SIGCHLD     18) SIGCONT     19) SIGSTOP     20) SIGTSTP
21) SIGTTIN     22) SIGTTOU     23) SIGURG      24) SIGXCPU     25) SIGXFSZ
26) SIGVTALRM   27) SIGPROF     28) SIGWINCH    29) SIGIO       30) SIGPWR
31) SIGSYS      34) SIGRTMIN    35) SIGRTMIN+1  36) SIGRTMIN+2  37) SIGRTMIN+3
38) SIGRTMIN+4  39) SIGRTMIN+5  40) SIGRTMIN+6  41) SIGRTMIN+7  42) SIGRTMIN+8
43) SIGRTMIN+9  44) SIGRTMIN+10 45) SIGRTMIN+11 46) SIGRTMIN+12 47) SIGRTMIN+13
48) SIGRTMIN+14 49) SIGRTMIN+15 50) SIGRTMAX-14 51) SIGRTMAX-13 52) SIGRTMAX-12
53) SIGRTMAX-11 54) SIGRTMAX-10 55) SIGRTMAX-9  56) SIGRTMAX-8  57) SIGRTMAX-7
58) SIGRTMAX-6  59) SIGRTMAX-5  60) SIGRTMAX-4  61) SIGRTMAX-3  62) SIGRTMAX-2
63) SIGRTMAX-1  64) SIGRTMAX

On AIX you would run:

$ kill -17 $SPID
$ kill -19 $SPID

$ kill -l
1) HUP                  14) ALRM                 27) MSG                  40) bad trap             53) bad trap
2) INT                  15) TERM                 28) WINCH                41) bad trap             54) bad trap
3) QUIT                 16) URG                  29) PWR                  42) bad trap             55) bad trap
4) ILL                  17) STOP                 30) USR1                 43) bad trap             56) bad trap
5) TRAP                 18) TSTP                 31) USR2                 44) bad trap             57) bad trap
6) ABRT                 19) CONT                 32) PROF                 45) bad trap             58) RECONFIG
7) EMT                  20) CHLD                 33) DANGER               46) bad trap             59) CPUFAIL
8) FPE                  21) TTIN                 34) VTALRM               47) bad trap             60) GRANT
9) KILL                 22) TTOU                 35) MIGRATE              48) bad trap             61) RETRACT
10) BUS                 23) IO                   36) PRE                  49) bad trap             62) SOUND
11) SEGV                24) XCPU                 37) VIRT                 50) bad trap             63) SAK
12) SYS                 25) XFSZ                 38) ALRM1                51) bad trap
13) PIPE                26) bad trap             39) WAITING              52) bad trap

Be very careful, different UNIX systems have different mappings between the signal number and the signal itself. Make sure you look it up first with “kill -l” to get the correct one. From my point of view, the suspend/resume feature, either with the Oracle oradebug or the UNIX kill command  is very useful.

Cheers,
William

Cet article Oracle – Suspending/Resuming an Oracle PID with “oradebug suspend/resume” or the OS kill command est apparu en premier sur Blog dbi services.

Documentum story. What is behind this story?
Normally I should write Documentum Platform story. Indeed dbi services is involved to build and manage a worldwide Documentum Platform. Our part covers all middleware components related to the infrastructure excepting the OS, DB and applications which are managed by other teams.

So we installed and manage so far:
– 30+ Documentum Servers, 45+ repositories
– 25+ Fulltext Servers,  50+ IndexAgent
– 15+ ADTS servers
– 30+ WebLogic servers, 30+ Domains, 70+ Managed Servers
– 15+ DA and the Content Server part for D2.

As the projects started two years ago, we also had to do some upgrades, patches, hotfixes during this phase.
Why patches and hotfixes? Let me think… Oh yeah to solve issues!

In the next few weeks, a serie of blogs (all starting with Documentum story) will be published to explain how we solved issues, managed challenges we had to deal with and to provide you hint and tips on different subjects.
The project is not finished yet, we had lots to do and some new missions are coming but we think, it is now time to share our experience related to installation, SSL and security, xPlore, Thumbnail server and ADTS, monitoring, WebLogic, SSO, performances, jobs etc.

Hopefully you will learn something with our posts.

Cet article Documentum story – Build and manage a Documentum Platform est apparu en premier sur Blog dbi services.

During a recent project at one of our customers, we often saw the message “Unexpected errors occurred while installing DARs”. In our case, this message happened when installing, migrating or upgrading a docbase on an already existing Content Server. We never saw this message during the first initial phase of installation of our repositories but we started to see it some months later with the first migration/upgrade. In this blog I will show you where does this issue can come from and how DARs are managed by Documentum for new/migrated docbases. In a future blog I will show you a home-made script that can be used to manually install DARs on docbases which tips, aso…

For this blog, let’s use the following:

• Documentum CS 7.2
• RedHat Linux 6.6
• $DOCUMENTUM=/app/dctm/server
• $DM_HOME=/app/dctm/server/product/7.2

Most of the time, these errors are thrown because Documentum isn’t able to install the needed DARs but what’s the reason behind that? First of all, there is one important thing to know: when installing a new docbase, Documentum will check which DARs should be installed by default. This list is dynamically generated based on a file and this file is the following one:

[dmadmin@content_server_01 ~]$ cat $DM_HOME/install/darsAdditional.xml
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<actions>
    <dar name="TCMReferenceProject">
        <description>TCMReferenceProject</description>
        <darFile>/app/dctm/server/product/7.2/install/DARsInternal/TCMReferenceProject.dar</darFile>
    </dar>
    <dar name="Forms">
        <description>Forms</description>
        <darFile>/app/dctm/server/product/7.2/install/DARsInternal/Forms.dar</darFile>
    </dar>
    <dar name="Collaboration Services">
        <description>Collaboration Services</description>
        <darFile>/app/dctm/server/product/7.2/install/DARsInternal/Collaboration Services.dar</darFile>
        <javaOptions>
            <javaOption>-XX:MaxPermSize=256m</javaOption>
            <javaOption>-Xmx1024m</javaOption>
        </javaOptions>
    </dar>
    <dar name="xcp">
        <description>xCP</description>
        <darFile>/app/dctm/server/product/7.2/install/DARsInternal/xcp.dar</darFile>
        <javaOptions>
            <javaOption>-XX:MaxPermSize=256m</javaOption>
            <javaOption>-Xmx1024m</javaOption>
        </javaOptions>
    </dar>
    <dar name="bpm">
        <description>BPM</description>
        <darFile>/app/dctm/server/product/7.2/install/DARsInternal/BPM.dar</darFile>
        <javaOptions>
            <javaOption>-XX:MaxPermSize=256m</javaOption>
            <javaOption>-Xmx1024m</javaOption>
        </javaOptions>
    </dar>
    <dar name="C2-DAR">
        <description>C2-DAR</description>
        <darFile>/app/dctm/server/product/7.2/install/DARsInternal/C2-DAR.dar</darFile>
        <javaOptions>
           <javaOption>-XX:MaxPermSize=256m</javaOption>
            <javaOption>-Xmx1024m</javaOption>
        </javaOptions>
    </dar>
    <dar name="D2-DAR">
        <description>D2-DAR</description>
        <darFile>/app/dctm/server/product/7.2/install/DARsInternal/D2-DAR.dar</darFile>
        <javaOptions>
            <javaOption>-XX:MaxPermSize=256m</javaOption>
            <javaOption>-Xmx1024m</javaOption>
        </javaOptions>
    </dar>
    <dar name="D2Widget-DAR">
        <description>D2Widget-DAR</description>
        <darFile>/app/dctm/server/product/7.2/install/DARsInternal/D2Widget-DAR.dar</darFile>
        <javaOptions>
            <javaOption>-XX:MaxPermSize=256m</javaOption>
            <javaOption>-Xmx1024m</javaOption>
        </javaOptions>
    </dar>
    <dar name="D2-Bin-DAR">
        <description>D2-Bin-DAR</description>
        <darFile>/app/dctm/server/product/7.2/install/DARsInternal/D2-Bin-DAR.dar</darFile>
        <javaOptions>
            <javaOption>-XX:MaxPermSize=256m</javaOption>
            <javaOption>-Xmx1024m</javaOption>
        </javaOptions>
    </dar>
    <dar name="O2-DAR">
        <description>O2-DAR</description>
        <darFile>/app/dctm/server/product/7.2/install/DARsInternal/O2-DAR.dar</darFile>
        <javaOptions>
            <javaOption>-XX:MaxPermSize=256m</javaOption>
            <javaOption>-Xmx1024m</javaOption>
        </javaOptions>
    </dar>
</actions>

Some of the dars inside this file are configured by Documentum directly (BPM & xcp DARs) and some others have been added by us, manually (D2 DARs). If you want to install new DARs for future installations of a docbase, then you can just update this file with a new section using this template:

    <dar name="DAR_NAME">
        <description>DAR_NAME</description>
        <darFile>/ABSOLUTE_LOCATION_OF_FILE/DAR_NAME.dar</darFile>
        <javaOptions>
            <javaOption>-XX:MaxPermSize=256m</javaOption>
            <javaOption>-Xmx1024m</javaOption>
        </javaOptions>
    </dar>

By default Documentum will always put the DARs inside the folder $DM_HOME/install/DARsInternal/ so I would recommend you to do the same, update the xml file and that’s it. Now this *can* also bring some trouble where some DARs aren’t installed anymore with the error shown at the beginning of this blog and the reason for that is – most of time – simply because there is a space in the name of the DAR… Yes from time to time, depending on the DARs, Documentum might not be able to properly manage spaces in the name of the DARs. It doesn’t always happen and that’s the annoying part because I didn’t find any logical behavior.

There is a way to verify which DAR might cause this issue and which one will not: when installing a CS patch, the folder “$DOCUMENTUM/patch/bin” is usually created and inside this folder, there is a file named “repositoryPatch.sh”. This script will be used by the patch to do some work and to install some DARs if needed. The interesting thing here is that this script include a small bug which you can use to find the troublesome DARs and you can also easily fix the script. After doing that, you will be able to use this script for all DARs no matter if they include spaces or not. So let’s take a look at the default file in one of our Content Server:

[dmadmin@content_server_01 ~]$ cat $DOCUMENTUM/patch/bin/repositoryPatch.sh | grep "^dars"
dars="/app/dctm/server/product/7.2/install/DARsInternal/LDAP.dar,/app/dctm/server/product/7.2/install/DARsInternal/MessagingApp.dar,/app/dctm/server/product/7.2/install/DARsInternal/MailApp.dar,/app/dctm/server/product/7.2/install/DARsInternal/Extended Search - SearchTemplates.dar,/app/dctm/server/product/7.2/install/DARsInternal/ATMOS Plugin.dar,/app/dctm/server/product/7.2/install/DARsInternal/VIPR Plugin.dar"

As you can see above, you just need to define the full path of each DAR file separated by a comma. To fix this script for all DARs, a first solution would be to rename the DARs but there is actually a simpler solution: use single quotes instead of double quotes in the dars definition:

[dmadmin@content_server_01 ~]$ cat $DOCUMENTUM/patch/bin/repositoryPatch.sh | grep "^dars"
dars='/app/dctm/server/product/7.2/install/DARsInternal/LDAP.dar,/app/dctm/server/product/7.2/install/DARsInternal/MessagingApp.dar,/app/dctm/server/product/7.2/install/DARsInternal/MailApp.dar,/app/dctm/server/product/7.2/install/DARsInternal/Extended Search - SearchTemplates.dar,/app/dctm/server/product/7.2/install/DARsInternal/ATMOS Plugin.dar,/app/dctm/server/product/7.2/install/DARsInternal/VIPR Plugin.dar'

By doing that, you corrected the bug in the script and now you should be able to execute this script to deploy all DARs to a single repository using:

[dmadmin@content_server_01 ~]$ $DOCUMENTUM/patch/bin/repositoryPatch.sh DOCBASE USERNAME PASSWORD

Note: As always, if you are using the Installation Owner as the USERNAME, then the PASSWORD can be a dummy password like “xxx” since there is the local trust on the Content Server.

This conclude this blog about the principal issue that we can face when installing a DAR, about how to manage the automatic deployment of some DARs in new docbases and finally how to use the script provided by a patch to do that manually. See you!

Cet article Documentum story – Management of DARs and unexpected errors est apparu en premier sur Blog dbi services.

Tim Hall had the idea that as many people as possible would write a small blog post about their favorite Oracle feature and we all post them on the same day. Here is my favorite feature that I described at the “EOUC Database ACES Share Their Favorite Database Things“: Transportable Tablespaces appeared in Oracle 8.1.5

ROWID

I’ll start with a change that came between Oracle 7 and Oracle 8. The ROWID, which identifies the physical location of a row within an database (with file ID, block offset, and row directory number) changed to be the location within a tablespace only. The format did not change, but the file ID was changed to be a relative file number instead of an absolute file number.
Here is the idea:

Actually, to be able to migrate without visiting each block (the ROWID is present in all blocks, all redo vectors, etc) they used the same number but that number is unique only within a tablespace. The first goal was to hold more datafiles per tablespace (Oracle 8 was the introduction of VLDB – Very Large Databases – concepts). The limit of 255 datafiles per database became a limit of 255 datafiles per tablespace. So the numbers starts the same as before but can go further.

This change was simple because anytime you want to fetch a row by its ROWID you know which table you query, so you know the tablespace. The exception is when the ROWID comes from a global index on a partitioned table, and for this case Oracle 8 introduced an extended ROWID that contains additional bytes to identify the segment by its DATA_OBJECT_ID.

By the way, this makes tablespaces more independent on the database that contains them because all row addressing is relative.

Locally Managed Tablespaces

Another change in 8i was Locally Managed Tablespaces. Before, the space management of the tablespaces was centralized in the database dictionary. Now, it is delocalized in each tablespace. What was stored in UET$ system table is now managed as a bitmap in the first datafile header.

Pluggable tablespaces

The original name of transportable tablespace was “pluggable tablespaces”. Because they are now more self-contained, you can detach them from a database an attach them to another database, without changing the content. This means that data is moved physically which is faster than the select/inserts that are behind a logical export/import. There are only two things that do not come with the datafiles.

The open transactions store their undo in the database UNDO tablespace. This means that if you detach a user tablespace you don’t have the information to rollback the ongoing transactions when you re-attach it elsewhere. For this reason, this ‘detach’ is possible only when there are no on-going transactions: you have to put the tablespace READ ONLY.

The user object metadata is stored in the database dictionary. Without them, the datafiles is just a bunch of bytes. You need the metadata to know what is a table or index, and which one. So, with transportable tablespaces, a logical export/import remains for the metadata only. This was done with exp/imp when TTS were introduced and is now done with DataPump. Small metadata is moved logically. Large data is moved physically.

Transportable tablespaces

TTS is faster than simple DataPump because data is moved physically by moving the datafiles. TTS is more flexible than an RMAN duplicate because you can move a subset of a database easily. Because the metadata is still transported logically, and datafiles are compatible with newer versions, TTS can be done cross-version, which makes it a nice way to migrate and upgrade. It is used also for tablespace point-in-time recovery where you have to recover to an auxiliary instance and then transport the tablespace to the target.
TTS is also used to move data quickly from operational database to a datawarehouse ODS.
It is also a good way to publish and share a database in read-only, on a DVD for example.

And beyond

Except with the move to DataPump for the metadata transfer, TTS has not change a lot until 12c. In 12.1 you have full transportable tablespace which automates the operations when you want to move a whole database. This can be used to migrate from non-CDB to multitenant architecture.

With multitenant, pluggable databases is an extension of TTS. Because user metadata come with the PDB system tablespaces yon don’t need to export them logically anymore: you transport the whole PDB. That’s the first restriction relieved. The second restriction, the need for read only, will be relieved as well when the UNDO will be local to the PDB and I don’t think I disclose any secret when telling that local UNDO has been announced for 12.2

OTN Appreciation Day

This was my contribution to the “EOUC Database ACES Share Their Favorite Database Things” at Oracle Open World 2016 organized by Debra Lilley. Tim Hall idea of “OTN Appreciation Day” comes from that. You still have time to contribute for this day. No need for long posts – I always write a but more than what I plan to. The “rules” for this day is described in oracle-base.com

Cet article OTN Appreciation Day : Transportable tablespaces est apparu en premier sur Blog dbi services.

In a previous blog (click here), I presented a common issue that might occur during the installation of some DARs and how to handle that with what Documentum provides but there are some limitations to that. Indeed the script repositoryPatch.sh is pretty good (except the small bug explained in the other blog) but its execution is limited to only one docbase and it is pretty boring to always put the full path of the DARs file knowing that usually all DARs will be at the same place (or at least this is what I would recommend). In addition to that, this script repositoryPatch.sh might not be available in your Content Server because it is normally available only after applying a patch of the Content Server. Therefore we usually use our own shell script to deploy X DARs on Y docbases with a single command.

For this blog, let’s use the following:

• Documentum CS 7.2
• RedHat Linux 6.6
• $DOCUMENTUM=/app/dctm/server
• $DM_HOME=/app/dctm/server/product/7.2

I will propose you in this blog three different solutions to avoid the issue with the space in the name of a DAR and to be able to deploy all DARs that you want on all docbases that you define.

1. Variable with space separated list

#!/bin/sh
docbases="DOCBASE1 DOCBASE2 DOCBASE3"
dar_list=("DAR 1.dar" "DAR 2.dar" "DAR 3.dar")
username="INSTALL_OWNER"
password="xxx"
dar_location="/app/dctm/server/product/7.2/install/DARsInternal"
 
for docbase in $docbases
do
        for dar in "${dar_list[@]}"
        do
                darname=${dar##*/}
 
                echo "Deploying $darname into $docbase"
                ts=$(date "+%Y%m%d-%H%M%S")
 
                $JAVA_HOME/bin/java -Ddar="$dar_location/$dar" \
                        -Dlogpath="$dar_location/dar-deploy-$darname-$docbase-$ts.log" \
                        -Ddocbase=$docbase -Duser=$username -Ddomain= -Dpassword="$password" \
                        -cp $DM_HOME/install/composer/ComposerHeadless/startup.jar \
                        org.eclipse.core.launcher.Main \
                        -data $DM_HOME/install/composer/workspace \
                        -application org.eclipse.ant.core.antRunner \
                        -buildfile $DM_HOME/install/composer/deploy.xml
        done
done

This is probably not the best solution because you have to manually add double quotes around each DAR name so that’s a little bit boring, unless you already have such a list. Please note that with this script, all DARs must be in the folder $DM_HOME/install/DARsInternal/ which is the folder used by Documentum by default for DARs.

2. No variable but still space separated list

#!/bin/sh
docbases="DOCBASE1 DOCBASE2 DOCBASE3"
username="INSTALL_OWNER"
password="xxx"
dar_location="/app/dctm/server/product/7.2/install/DARsInternal"
 
for docbase in $docbases
do
        for dar in "DAR 1.dar" "DAR 2.dar" "DAR 3.dar"
        do
                darname=${dar##*/}
 
                echo "Deploying $darname into $docbase"
                ts=$(date "+%Y%m%d-%H%M%S")
 
                $JAVA_HOME/bin/java -Ddar="$dar_location/$dar" \
                        -Dlogpath="$dar_location/dar-deploy-$darname-$docbase-$ts.log" \
                        -Ddocbase=$docbase -Duser=$username -Ddomain= -Dpassword="$password" \
                        -cp $DM_HOME/install/composer/ComposerHeadless/startup.jar \
                        org.eclipse.core.launcher.Main \
                        -data $DM_HOME/install/composer/workspace \
                        -application org.eclipse.ant.core.antRunner \
                        -buildfile $DM_HOME/install/composer/deploy.xml
        done
done

Same as before for this one, you don’t need the @ trick since the list of DARs is in the for loop directly but you still need to manually put double quotes around the file names.

3. Variable with comma separated list

#!/bin/sh
docbases="DOCBASE1 DOCBASE2 DOCBASE3"
dar_list="DAR 1.dar,DAR 2.dar,DAR 3.dar"
username="INSTALL_OWNER"
password="xxx"
dar_location="/app/dctm/server/product/7.2/install/DARsInternal"
 
for docbase in $docbases
do
        IFS=',' ; for dar in $dar_list
        do
                darname=${dar##*/}
 
                echo "Deploying $darname into $docbase"
                ts=$(date "+%Y%m%d-%H%M%S")
 
                $JAVA_HOME/bin/java -Ddar="$dar_location/$dar" \
                        -Dlogpath="$dar_location/dar-deploy-$darname-$docbase-$ts.log" \
                        -Ddocbase=$docbase -Duser=$username -Ddomain= -Dpassword="$password" \
                        -cp $DM_HOME/install/composer/ComposerHeadless/startup.jar \
                        org.eclipse.core.launcher.Main \
                        -data $DM_HOME/install/composer/workspace \
                        -application org.eclipse.ant.core.antRunner \
                        -buildfile $DM_HOME/install/composer/deploy.xml
        done
done

This version is my preferred one because what you need is just a list of all DARs to be installed and the separation is just a comma so that’s pretty simple to obtain and simpler to manage than double quotes everywhere. Now these versions will all provide the following output showing that the script is working properly even for DARs containing spaces in their names:

Deploying DAR 1.dar into DOCBASE1
Deploying DAR 2.dar into DOCBASE1
Deploying DAR 3.dar into DOCBASE1
Deploying DAR 1.dar into DOCBASE2
Deploying DAR 2.dar into DOCBASE2
Deploying DAR 3.dar into DOCBASE2
Deploying DAR 1.dar into DOCBASE3
Deploying DAR 2.dar into DOCBASE3
Deploying DAR 3.dar into DOCBASE3

So that was for the deployment of several DARs in several docbases. By default Documentum will consider that the username is “dmadmin”. If this isn’t the case, then this script will not work in its current state. Yes I know, we specified the user in the script but Documentum doesn’t care and it will fail if you aren’t using dmadmin. If you need to specify another name for the Installation Owner, then you need to do three additional things. The first one is to add a new parameter in the script that will therefore now look like the following:

#!/bin/sh
docbases="DOCBASE1 DOCBASE2 DOCBASE3"
dar_list="DAR 1.dar,DAR 2.dar,DAR 3.dar"
username="INSTALL_OWNER"
password="xxx"
dar_location="/app/dctm/server/product/7.2/install/DARsInternal"
 
for docbase in $docbases
do
        IFS=',' ; for dar in $dar_list
        do
                darname=${dar##*/}
 
                echo "Deploying $darname into $docbase"
                ts=$(date "+%Y%m%d-%H%M%S")
 
                $JAVA_HOME/bin/java -Ddar="$dar_location/$dar" \
                        -Dlogpath="$dar_location/dar-deploy-$darname-$docbase-$ts.log" \
                        -Ddocbase=$docbase -Duser=$username -Ddomain= -Dpassword="$password" \
                        -Dinstallparam="$dar_location/installparam.xml" \
                        -cp $DM_HOME/install/composer/ComposerHeadless/startup.jar \
                        org.eclipse.core.launcher.Main \
                        -data $DM_HOME/install/composer/workspace \
                        -application org.eclipse.ant.core.antRunner \
                        -buildfile $DM_HOME/install/composer/deploy.xml
        done
done

After doing that, the second thing to do is to create the file installparam.xml that we used above. In this case, I put this file in $DM_HOME/install/DARsInternal but you can put it wherever you want.

[dmadmin@content_server_01 ~]$ cat $DM_HOME/install/DARsInternal/installparam.xml
<?xml version="1.0" encoding="UTF-8"?>
<installparam:InputFile xmlns:installparam="installparam" xmlns:xmi="http://www.omg.org/XMI" xmi:version="2.0">
    <parameter value="dmadmin" key="YOUR_INSTALL_OWNER"/>
</installparam:InputFile>

Just replace in this file YOUR_INSTALL_OWNER with the name of your Installation Owner. Finally the last thing to do is to update the buildfile. In our script, we are using the default one provided by EMC. In this buildfile, you need to specifically tell Documentum that you want it to take into account a custom parameter file and this is done by adding a single line in the emc.install XML tag:

[dmadmin@content_server_01 ~]$ grep -A5 emc.install $DM_HOME/install/composer/deploy.xml
        <emc.install dar="${dar}"
                     docbase="${docbase}"
                     username="${user}"
                     password="${password}"
                     domain="${domain}"
                     inputfile="${installparam}" />

Once this is done, you can just restart the deployment of DARs and it should be successful this time. Another solution to specify another Installation Owner or add more install parameters is to not use the default buildfile provided by EMC but use your own custom buildile. This will be an ANT file (xml with project, target, aso…) that will define what to do exactly so this is highly customizable. So yeah there are a lot of possibilities!

Note: Once done, don’t forget to remove the line from the file deploy.xml

Hope you enjoyed this blog and that this will give you some ideas about how to improve your processes or how to do more with less. See you soon!

Cet article Documentum story – Manual deployment of X DARs on Y docbases est apparu en premier sur Blog dbi services.

RMAN persistent settings can be managed in two different ways.

• Via the RMAN interface
  – e.g. RMAN> CONFIGURE BACKUP OPTIMIZATION ON;
• Via SQL
  – e.g. VARIABLE RECNO NUMBER;
  EXECUTE :RECNO := SYS.DBMS_BACKUP_RESTORE.SETCONFIG(‘BACKUP OPTIMIZATION’,’ON’);

There are several scenarios when it might be helpful to use the SQL way. I will show 3 of them:

• Automation
• Reset to default
• Rebuilding the RMAN persistent settings after losing all controlfiles (no catalog)

Let’s take a look at the first scenario. For example, when you have an automated way to run SQL’s against all of your databases and you want to change the RMAN retention from 3 days to 4 days for all of your databases. Then you could run the following.

SQL> select conf#, name, value from v$rman_configuration where name = 'RETENTION POLICY';

CONF# NAME                             VALUE
----- -------------------------------- ----------------------------------------------------------------------------------------
    1 RETENTION POLICY                 TO RECOVERY WINDOW OF 3 DAYS


SQL> EXECUTE DBMS_BACKUP_RESTORE.DELETECONFIG(CONF# => 1);

PL/SQL procedure successfully completed.

SQL> VARIABLE RECNO NUMBER;
SQL> EXECUTE :RECNO := SYS.DBMS_BACKUP_RESTORE.SETCONFIG('RETENTION POLICY','TO RECOVERY WINDOW OF 4 DAYS');

PL/SQL procedure successfully completed.


SQL> select conf#, name, value from v$rman_configuration where name = 'RETENTION POLICY';

CONF# NAME                             VALUE
----- -------------------------------- ----------------------------------------------------------------------------------------
    1 RETENTION POLICY                 TO RECOVERY WINDOW OF 4 DAYS

	
-- The new value is, of course, immediately reflected via the RMAN interface as well

RMAN> SHOW RETENTION POLICY;

RMAN configuration parameters for database with db_unique_name OCM121 are:
CONFIGURE RETENTION POLICY TO RECOVERY WINDOW OF 4 DAYS;

The second useful scenario might be, to reset the whole RMAN config with one shot, instead of running several clear commands like the following, “RMAN> CONFIGURE BACKUP OPTIMIZATION CLEAR;” , simply run the RESETCONFIG.

SQL> EXECUTE DBMS_BACKUP_RESTORE.RESETCONFIG;

PL/SQL procedure successfully completed.

-- After executing this command, the v$rman_configuration view is empty, which means that all
-- RMAN persistent settings are default.

SQL> select conf#, name, value from v$rman_configuration;

no rows selected

And last but not least, to restore the RMAN persistent settings via SQL, in case you have lost all of your controlfiles and no RMAN catalog is in place.

One little side note, in case you have a RMAN catalog. The RMAN sync from the controlfile to the catalog is usually unidirectional, meaning that the controlfile is always the master and it syncs the information to the catalog. However, there are exceptions were it is bidirectional. One of it is, when you recreate the controlfile manually, then RMAN is able to get the last RMAN persistent settings from the catalog and applies it to the controlfile.

However, if you don’t have a catalog, dump out the RMAN persistent settings into SQL, simply by backing up the controlfile to trace.

SQL> alter database backup controlfile to trace as '/tmp/cntrl.trc';

Database altered.

-- Configure RMAN configuration record 1
VARIABLE RECNO NUMBER;
EXECUTE :RECNO := SYS.DBMS_BACKUP_RESTORE.SETCONFIG('RETENTION POLICY','TO RECOVERY WINDOW OF 4 DAYS');
-- Configure RMAN configuration record 2
VARIABLE RECNO NUMBER;
EXECUTE :RECNO := SYS.DBMS_BACKUP_RESTORE.SETCONFIG('BACKUP OPTIMIZATION','ON');
-- Configure RMAN configuration record 3
VARIABLE RECNO NUMBER;
EXECUTE :RECNO := SYS.DBMS_BACKUP_RESTORE.SETCONFIG('CONTROLFILE AUTOBACKUP','ON');
-- Configure RMAN configuration record 4
VARIABLE RECNO NUMBER;
EXECUTE :RECNO := SYS.DBMS_BACKUP_RESTORE.SETCONFIG('RMAN OUTPUT','TO KEEP FOR 14 DAYS');
-- Configure RMAN configuration record 5
VARIABLE RECNO NUMBER;
EXECUTE :RECNO := SYS.DBMS_BACKUP_RESTORE.SETCONFIG('DEVICE TYPE','DISK PARALLELISM 2 BACKUP TYPE TO BACKUPSET');
-- Configure RMAN configuration record 6
VARIABLE RECNO NUMBER;
EXECUTE :RECNO := SYS.DBMS_BACKUP_RESTORE.SETCONFIG('DEFAULT DEVICE TYPE TO','DISK');
-- Configure RMAN configuration record 7
VARIABLE RECNO NUMBER;
EXECUTE :RECNO := SYS.DBMS_BACKUP_RESTORE.SETCONFIG('CHANNEL','DEVICE TYPE ''SBT_TAPE'' PARMS  ''SBT_LIBRARY=oracle.disksbt,ENV=(BACKUP_DIR=/nfs/OCM121)''');
-- Configure RMAN configuration record 8
VARIABLE RECNO NUMBER;
EXECUTE :RECNO := SYS.DBMS_BACKUP_RESTORE.SETCONFIG('DEVICE TYPE','''SBT_TAPE'' PARALLELISM 2 BACKUP TYPE TO COMPRESSED BACKUPSET');
-- Configure RMAN configuration record 9
VARIABLE RECNO NUMBER;
EXECUTE :RECNO := SYS.DBMS_BACKUP_RESTORE.SETCONFIG('ARCHIVELOG DELETION POLICY','TO APPLIED ON ALL STANDBY');

And if you run into the severe situation of losing all controlfiles, you can restore the RMAN persistent settings quite quickly. Especially useful, when you have configured complex Media Manager settings.
Cheers,
William

P.S. Managing RMAN persistent settings via SQL is not a 12c feature. It exists for quite a long time.

Cet article Oracle 12c – Managing RMAN persistent settings via SQL est apparu en premier sur Blog dbi services.

Tim Hall had the idea that as many people as possible would write a small blog post about their favorite Oracle feature and we all post them on the same day. Here is my favorite feature: ADVM – The Oracle ASM Dynamic Volume Manager.

So, what is it? The docs tell you this: “Oracle ASM Dynamic Volume Manager (Oracle ADVM) provides volume management services and a standard disk device driver interface to clients. File systems and other disk-based applications send I/O requests to Oracle ADVM volume devices as they would to other storage devices on a vendor operating system.”

The easy to understand version is this: It enables us to use regular file systems on top of ASM.

Does is make sense to use it? When you have ASM running on the host or all the hosts of a Grid Infrastructure cluster anyway then it definitely makes sense. ASM will do all the mirroring and striping for you so there is no need to use another technology to achieve that when you can create ADVM volumes and create file systems on top of these. Although the most common scenario is to create an ACFS file system on top of the volumes you are actually not limited to that. Lets do a short demo.

Lets say we have these devices available for use by the grid user:

[root@rac1 ~] ls -la /dev/sd[b-f]*
brw-rw----. 1 root disk     8, 16 Oct 10 17:54 /dev/sdb
brw-rw----. 1 grid asmadmin 8, 17 Oct 10 18:10 /dev/sdb1
brw-rw----. 1 root disk     8, 32 Oct 10 17:54 /dev/sdc
brw-rw----. 1 grid asmadmin 8, 33 Oct 10 18:10 /dev/sdc1
brw-rw----. 1 root disk     8, 48 Oct 10 17:54 /dev/sdd
brw-rw----. 1 grid asmadmin 8, 49 Oct 10 18:10 /dev/sdd1
brw-rw----. 1 root disk     8, 64 Oct 10 17:54 /dev/sde
brw-rw----. 1 grid asmadmin 8, 65 Oct 10 18:10 /dev/sde1
brw-rw----. 1 root disk     8, 80 Oct 10 17:54 /dev/sdf
brw-rw----. 1 grid asmadmin 8, 81 Oct 10 18:10 /dev/sdf1

We want to use “/dev/sde1″ for our new ADVM volume. What we need is an ASM diskgroup in a first step because for creating an ADVM volume you’ll need a ASM diskgroup where you can place your volume on:

grid@rac1:/home/grid/ [+ASM1] sqlplus / as sysasm
SQL> create diskgroup ADVM external redundancy disk '/dev/sde1';

Diskgroup created.

SQL> 

Ok, fine. How can we proceed with creating a volume? Quite easy:

grid@rac1:/home/grid/ [+ASM1] asmcmd volcreate -G ADMV -s 2g VOLADVM
ORA-15032: not all alterations performed
ORA-15221: ASM operation requires compatible.asm of 11.2.0.0.0 or higher (DBD ERROR: OCIStmtExecute)

Hm, quite clear when you search the documentation: ADVM is available since 11gR2:

Easy to fix:

grid@rac1:/home/grid/ [+ASM1] sqlplus / as sysasm

SQL> alter diskgroup ADVM set attribute 'compatible.asm'='12.1';

Diskgroup altered.

SQL> 

Lets try again:

grid@rac1:/home/grid/ [+ASM1] asmcmd volcreate -G ADMV -s 2g VOLADVM
grid@rac1:/home/grid/ [+ASM1] 

Perfect. Now I have a volume visible to the operating system:

grid@rac1:/home/grid/ [+ASM1] ls -la /dev/asm/*advm*
brwxrwx---. 1 root asmadmin 252, 115201 Oct 10 18:20 /dev/asm/voladvm-225

On top of this volume we can now create file systems. The natural one would be ACFS:

[root@rac1 ~] mkfs.acfs /dev/asm/voladvm-225
mkfs.acfs: version                   = 12.1.0.2.0
mkfs.acfs: on-disk version           = 39.0
mkfs.acfs: volume                    = /dev/asm/voladvm-225
mkfs.acfs: volume size               = 2147483648  (   2.00 GB )
mkfs.acfs: Format complete.

But in fact every other file system the operating system supports is possible, too:

[root@rac1 ~] mkfs.xfs /dev/asm/voladvm-225
meta-data=/dev/asm/voladvm-225   isize=256    agcount=4, agsize=131072 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=0        finobt=0
data     =                       bsize=4096   blocks=524288, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=0
log      =internal log           bsize=4096   blocks=2560, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

Quite cool, isn’t it? Whatever file system your operating system supports can be put on ASM disk groups …

Cet article OTN Appreciation Day : ADVM est apparu en premier sur Blog dbi services.