Oracle RAC i.e. Real Application Cluster is a very unique database infrastructure. I have seen the term ‘cluster’ in other database world, but they are usually the “cluster” of both servers and storages ( for example MariaDB Galera cluster). In Oracle, the database is defined as a collection of physical operating system files or disks. Instance is a set of Oracle background processes or threads running on the same computer(aka node, server) and sharing a common memory(i.e. SGA, System Global Area). For quite a time ago, we can have at most one instance per database. We can think of a database as an island (lots of treasure there). Instance is a bridge with a particular traffic control system from the mainland to this island. An Oracle RAC database then can be thought as that we have multiple bridges built to connect to the same island.

At a client side, we created and configured dozens of such 4-node (bridge) RAC databases. Normally we have a number with the name of the “bridge”, ie bridge 1, 2, 3,4 and the “traffic control systems (instances)” also have a number in their names, i.e. 1,2,3,4. Naturally we want to have instance 1 running on node 1, so on and so forth. However we see instance 3 running on node 4 in many cases for some reason unknown( not be able to trace the origin of that so far). This could be very inconvenient and error-prone ( i.e. may cause mismatch/confusion in some configuration files which contain database instance info, for example oratab file).

SRVCTL is the command line toot that can be used to manage configuration information of a RAC. We can use SRVCTL commands to add, remove, start, stop, modify, enable, and disable a number of entities, such as databases, instances, listeners, SCAN listeners, services, grid naming service (GNS), and Oracle ASM.

Below I summarize what we’ve done with SRVCTL commands to re-configure the RAC so that the node number and instance number become aligned.

-- get the current database status
$> srvctl status database -d oradb

Instance oradb1 is running on node oranode01
Instance oradb2 is running on node oranode02
Instance oradb3 is running on node oranode04
Instance oradb4 is running on node oranode03

-- remove instance 3 and 4
-- -f force option is needed otherwise failed
$> srvctl remove instance -d oradb -i oradb3 -f
$> srvctl remove instance -d oradb -i oradb4 -f

-- add back instance with desided node
$> srvctl add instance -d oradb -i oradb3 -n oranode03
$> srvctl add instance -d oradb -i oradb4 -n oranode04

-- start instance
$> srvctl start instance -d oradb -i oradb3
$> srvctl start instance -d oradb -i oradb4

-- verify the dabase status

$> srvctl status database -d oradb
Instance oradb1 is running on node oranode01
Instance oradb2 is running on node oranode02
Instance oradb3 is running on node oranode03
Instance oradb4 is running on node oranode04