ORA-27102: Out of Memory - Fix Linux Shared Memory Limits

ORA-27102: Out of Memory

Error Overview

Error Text: ORA-27102: out of memory

ORA-27102 is a critical OS-level error that occurs when Oracle cannot allocate sufficient memory for the System Global Area (SGA) or other memory structures during database startup or operation. Unlike ORA-04031 (which relates to in-database shared pool allocation), ORA-27102 indicates that the operating system itself has refused the memory allocation request. It most commonly appears on Linux systems where kernel shared memory parameters are not correctly sized for the Oracle instance.

Common Causes

1. Kernel SHMMAX Parameter Too Small

• kernel.shmmax defines the maximum size of a single shared memory segment
• Oracle requires one contiguous shared memory segment at least as large as the SGA
• If SHMMAX is smaller than the configured SGA size, the shmget() system call fails immediately
• Common default values on older Linux installations (32MB–256MB) are far too small for production Oracle

2. Kernel SHMALL Parameter Exhausted

• kernel.shmall controls the total number of shared memory pages system-wide
• When SHMALL is exhausted by existing allocations, no additional pages can be granted
• Multiple Oracle instances on the same host compete for this pool
• Other applications (PostgreSQL, Java app servers, Redis) also consume SHMALL pages

3. Insufficient SHMMNI (Shared Memory Segment Count)

• kernel.shmmni limits the total number of shared memory segments on the system
• On systems running many Oracle instances or many background processes, this limit can be reached
• The default of 4096 is usually adequate but can be exhausted in dense environments

4. memlock Limit Blocks HugePages Allocation

• When HugePages are configured, Oracle locks huge pages into physical RAM using mlock()
• The memlock ulimit for the oracle OS user must be large enough to cover the SGA
• If /etc/security/limits.conf has an insufficient memlock value, Oracle cannot pin memory
• This manifests as ORA-27102 even when kernel.shmmax appears correct

5. HugePages Misconfiguration

• HugePages not allocated (vm.nr_hugepages = 0) while use_large_pages = ONLY is set
• HugePages pool smaller than the SGA size
• Oracle user not in the group defined by vm.hugetlb_shm_group
• use_large_pages = ONLY fails hard rather than falling back to normal pages

6. System Memory Overcommit

• Physical RAM plus swap is insufficient for the requested allocation
• vm.overcommit_memory = 2 combined with a low vm.overcommit_ratio can deny allocations
• Other processes have consumed available memory since last boot
• OOM killer has pre-emptively restricted allocations

Diagnostic Queries

Check Oracle Memory Parameters

-- Review all memory-related parameters
SELECT name, value, description
FROM v$parameter
WHERE name IN (
  'sga_target', 'sga_max_size', 'memory_target', 'memory_max_target',
  'pga_aggregate_target', 'shared_pool_size', 'db_cache_size',
  'large_pool_size', 'java_pool_size', 'use_large_pages'
)
ORDER BY name;

-- Actual SGA component sizes
SELECT name,
       ROUND(value/1024/1024, 2)       AS size_mb,
       ROUND(value/1024/1024/1024, 2)  AS size_gb
FROM v$sga
ORDER BY value DESC;

-- Total SGA size needed
SELECT ROUND(SUM(value)/1024/1024/1024, 2) AS total_sga_gb
FROM v$sga;

Check HugePages Usage

-- Verify Oracle is using HugePages
SELECT name, value
FROM v$parameter
WHERE name = 'use_large_pages';

-- Check memory advisor for sizing recommendations
SELECT
  component,
  ROUND(current_size/1024/1024, 0)   AS current_mb,
  ROUND(min_size/1024/1024, 0)       AS min_mb
FROM v$sga_dynamic_components
ORDER BY current_size DESC;

Check System Memory State (OS Commands)

# Comprehensive kernel shared memory status
ipcs -lm

# Current shared memory segments (check sizes and owners)
ipcs -m

# Live kernel parameter values
sysctl kernel.shmmax kernel.shmall kernel.shmmni

# Compare to what Oracle needs
cat /proc/sys/kernel/shmmax
echo "In GB: $(echo "$(cat /proc/sys/kernel/shmmax) / 1024 / 1024 / 1024" | bc)"

# Total system memory
free -h
grep -E "(MemTotal|MemFree|MemAvailable|HugePages|Hugepagesize)" /proc/meminfo

# Oracle user limits (run as oracle OS user)
ulimit -a | grep -E "(virtual memory|locked)"

# Check memlock in limits.conf
grep -E "(oracle|oinstall|memlock)" /etc/security/limits.conf

Identify the Blocking Segment

# Find what is consuming shared memory
ipcs -m -u

# See per-segment details including permissions and size
ipcs -m -p

# Check if Oracle left orphaned segments from a previous crash
ipcs -m | grep oracle

# Get the exact error from the kernel
dmesg | grep -i "out of memory"
dmesg | tail -50
journalctl -k | grep -E "(oom|memory)" | tail -30

Step-by-Step Resolution

Step 1: Determine Required SGA Size

# Extract configured SGA from SPFILE when DB won't start
strings $ORACLE_HOME/dbs/spfile${ORACLE_SID}.ora | grep -iE "(sga_target|memory_target|sga_max)"

# Or check the init.ora / pfile
cat $ORACLE_HOME/dbs/init${ORACLE_SID}.ora | grep -iE "(sga|memory)"

Step 2: Set Correct Kernel Parameters

# Calculate the values you need
# SHMMAX must be >= SGA size (use SGA size in bytes)
# Example: 12GB SGA
SGA_BYTES=$((12 * 1024 * 1024 * 1024))
echo "Required SHMMAX (bytes): $SGA_BYTES"

# SHMALL must cover total shared memory (SGA + PGA headroom)
# SHMALL is in PAGES; default page size on x86-64 Linux is 4096 bytes
PAGE_SIZE=$(getconf PAGE_SIZE)
TOTAL_SHM_BYTES=$((16 * 1024 * 1024 * 1024))   # 16GB total budget
SHMALL_PAGES=$(( TOTAL_SHM_BYTES / PAGE_SIZE ))
echo "Required SHMALL (pages): $SHMALL_PAGES"

# Apply immediately (survives until next reboot)
sudo sysctl -w kernel.shmmax=$SGA_BYTES
sudo sysctl -w kernel.shmall=$SHMALL_PAGES
sudo sysctl -w kernel.shmmni=4096

Step 3: Make Kernel Parameters Permanent

# Back up existing configuration
sudo cp /etc/sysctl.conf /etc/sysctl.conf.bak.$(date +%Y%m%d)

# Add or update Oracle kernel parameters
sudo tee -a /etc/sysctl.conf > /dev/null << 'EOF'

# Oracle Database kernel parameters
kernel.shmmax = 12884901888    # 12GB - must be >= SGA size
kernel.shmall = 4194304        # 16GB in 4KB pages
kernel.shmmni = 4096           # Maximum shared memory segments
kernel.sem = 250 32000 100 128 # semmsl semmns semopm semmni
vm.overcommit_memory = 2
vm.overcommit_ratio = 80
EOF

# Apply permanently
sudo sysctl -p

# Verify
sysctl kernel.shmmax kernel.shmall kernel.shmmni

Step 4: Configure memlock and ulimits

# Edit /etc/security/limits.conf (requires root)
sudo tee -a /etc/security/limits.conf > /dev/null << 'EOF'

# Oracle user OS limits
oracle   soft   nofile    1024
oracle   hard   nofile    65536
oracle   soft   nproc     2047
oracle   hard   nproc     16384
oracle   soft   stack     10240
oracle   hard   stack     32768
oracle   soft   memlock   unlimited
oracle   hard   memlock   unlimited
EOF

# For systems using pam_limits, ensure it is loaded:
grep pam_limits /etc/pam.d/login
grep pam_limits /etc/pam.d/sshd

# Verify as the oracle OS user after a fresh login
su - oracle -c "ulimit -l"

Step 5: Configure HugePages (Recommended for Production)

# Get hugepage size
grep Hugepagesize /proc/meminfo

# Calculate pages needed (add 10% overhead)
SGA_GB=12
HUGEPAGE_SIZE_KB=$(grep Hugepagesize /proc/meminfo | awk '{print $2}')
HUGEPAGE_SIZE_MB=$(( HUGEPAGE_SIZE_KB / 1024 ))
HUGEPAGES_NEEDED=$(( (SGA_GB * 1024 / HUGEPAGE_SIZE_MB) * 110 / 100 ))
echo "HugePages needed: $HUGEPAGES_NEEDED"

# Get oracle group ID for hugetlb
getent group oinstall

# Set hugepages
sudo sysctl -w vm.nr_hugepages=$HUGEPAGES_NEEDED
sudo sysctl -w vm.hugetlb_shm_group=$(getent group oinstall | cut -d: -f3)

# Persist in sysctl.conf
echo "vm.nr_hugepages = $HUGEPAGES_NEEDED" | sudo tee -a /etc/sysctl.conf
echo "vm.hugetlb_shm_group = $(getent group oinstall | cut -d: -f3)" | sudo tee -a /etc/sysctl.conf
sudo sysctl -p

# Enable HugePages in Oracle (SPFILE)
sqlplus / as sysdba << 'EOF'
ALTER SYSTEM SET use_large_pages = TRUE SCOPE = SPFILE;
SHUTDOWN IMMEDIATE;
STARTUP;
EOF

# Confirm HugePages are allocated
grep -i hugepages /proc/meminfo

Step 6: Attempt Startup and Verify

# Start Oracle
sqlplus / as sysdba << 'EOF'
STARTUP;
EOF

# Confirm the SGA is allocated as a HugePages segment
ipcs -m | grep oracle
grep HugePages /proc/meminfo

Prevention Strategies

Validated sysctl.conf Template for Oracle

# /etc/sysctl.conf — Oracle 19c on Linux x86-64
# Adjust shmmax and shmall to match your SGA size

# Shared memory: SHMMAX >= SGA size in bytes
kernel.shmmax = 17179869184    # 16GB
kernel.shmall = 4194304        # 16GB / 4KB page = 4194304 pages
kernel.shmmni = 4096

# Semaphores: semmsl semmns semopm semmni
# Rule: semmsl >= PROCESSES parameter; semmns >= processes * 2
kernel.sem = 250 32000 100 128

# Memory overcommit: prevent OOM from granting impossible allocations
vm.overcommit_memory = 2
vm.overcommit_ratio = 80

# HugePages (set after calculating requirement)
vm.nr_hugepages = 8192         # For 16GB SGA with 2MB pages
vm.hugetlb_shm_group = 54321  # oinstall group GID

# Swap management
vm.swappiness = 10

Validated limits.conf Template for Oracle

# /etc/security/limits.conf — Oracle user limits
oracle   soft   nofile    1024
oracle   hard   nofile    65536
oracle   soft   nproc     2047
oracle   hard   nproc     16384
oracle   soft   stack     10240
oracle   hard   stack     32768
oracle   soft   memlock   unlimited
oracle   hard   memlock   unlimited

Pre-Startup Validation Script

#!/bin/bash
# oracle_memory_preflight.sh — Run before starting Oracle

SGA_GB=${1:-8}
SGA_BYTES=$(( SGA_GB * 1024 * 1024 * 1024 ))
PAGE_SIZE=$(getconf PAGE_SIZE)
PASS=0; FAIL=0

check() {
  local desc="$1"; local result="$2"
  if [ "$result" = "OK" ]; then
    echo "[OK]   $desc"; PASS=$((PASS+1))
  else
    echo "[FAIL] $desc — $result"; FAIL=$((FAIL+1))
  fi
}

SHMMAX=$(cat /proc/sys/kernel/shmmax)
[ $SHMMAX -ge $SGA_BYTES ] \
  && check "kernel.shmmax ($SHMMAX) >= SGA ($SGA_BYTES bytes)" "OK" \
  || check "kernel.shmmax ($SHMMAX) >= SGA ($SGA_BYTES bytes)" \
           "Set: sudo sysctl -w kernel.shmmax=$SGA_BYTES"

TOTAL_MEM_KB=$(grep MemTotal /proc/meminfo | awk '{print $2}')
TOTAL_MEM_BYTES=$(( TOTAL_MEM_KB * 1024 ))
[ $TOTAL_MEM_BYTES -gt $SGA_BYTES ] \
  && check "System RAM ($((TOTAL_MEM_BYTES/1024/1024/1024))GB) > SGA (${SGA_GB}GB)" "OK" \
  || check "System RAM sufficient" "Not enough physical RAM"

MEMLOCK=$(ulimit -l 2>/dev/null || echo 0)
[ "$MEMLOCK" = "unlimited" ] \
  && check "oracle memlock = unlimited" "OK" \
  || check "oracle memlock = unlimited" \
           "Current: $MEMLOCK — check /etc/security/limits.conf"

HP_TOTAL=$(grep HugePages_Total /proc/meminfo | awk '{print $2}')
HP_FREE=$(grep HugePages_Free /proc/meminfo | awk '{print $2}')
HP_SIZE_KB=$(grep Hugepagesize /proc/meminfo | awk '{print $2}')
HP_BYTES=$(( HP_FREE * HP_SIZE_KB * 1024 ))
[ $HP_BYTES -ge $SGA_BYTES ] \
  && check "Free HugePages (${HP_FREE} pages) >= SGA" "OK" \
  || check "Free HugePages (${HP_FREE} pages) >= SGA" \
           "Increase vm.nr_hugepages (currently $HP_TOTAL total)"

echo ""
echo "Results: $PASS passed, $FAIL failed"
[ $FAIL -eq 0 ] && exit 0 || exit 1

Proactive Monitoring Query

-- Detect memory pressure before it causes failures
SELECT
  name,
  ROUND(bytes/1024/1024, 2) AS size_mb,
  ROUND(bytes/1024/1024/1024, 2) AS size_gb
FROM v$sgainfo
WHERE name IN ('Maximum SGA Size', 'Total SGA Size', 'Free SGA Memory Available')
ORDER BY bytes DESC;

-- HugePages utilization from Oracle's perspective
SELECT
  value AS use_large_pages_setting
FROM v$parameter
WHERE name = 'use_large_pages';

Related Errors

• ORA-27125 - Unable to create shared memory segment (shmget failure)
• ORA-27123 - Unable to attach to shared memory segment (shmat failure)
• ORA-27140 - Attach to post/wait facility failed (semaphore limits)
• ORA-27300 - OS system dependent operation failed (generic OS error)
• ORA-00845 - MEMORY_TARGET not supported (AMM with /dev/shm)
• ORA-04031 - Unable to allocate shared memory within the SGA

Emergency Response

Quick Fixes

1. Apply kernel parameters without a reboot

   sudo sysctl -w kernel.shmmax=17179869184
   sudo sysctl -w kernel.shmall=4194304

2. Reduce SGA to fit within current OS limits

   sqlplus / as sysdba << 'EOF'
   ALTER SYSTEM SET sga_target = 6G SCOPE = SPFILE;
   ALTER SYSTEM SET memory_target = 0 SCOPE = SPFILE;
   STARTUP;
   EOF

3. Start with a PFILE to bypass SPFILE memory settings

   cp $ORACLE_HOME/dbs/spfile${ORACLE_SID}.ora /tmp/
   strings $ORACLE_HOME/dbs/spfile${ORACLE_SID}.ora | grep -v memory > /tmp/init_nomem.ora
   # Edit /tmp/init_nomem.ora to add conservative sga_target
   sqlplus / as sysdba
   STARTUP PFILE='/tmp/init_nomem.ora';

4. Remove orphaned shared memory segments from a crashed instance

   # Identify Oracle segments
   ipcs -m | grep oracle
   # Remove by shmid
   ipcrm -m <shmid>

Post-Resolution Steps

# Persist all kernel changes
sudo sysctl -p

# Confirm parameters are live
sysctl kernel.shmmax kernel.shmall kernel.shmmni

# Document the change
echo "$(date): ORA-27102 resolved — kernel.shmmax set to $(cat /proc/sys/kernel/shmmax)" \
  >> /var/log/oracle_changes.log