ORA-19502: Write Error on File - Fix RMAN Backup Failures

ORA-19502: Write Error on File, Block Number

Error Overview

Error Text: ORA-19502: write error on file "string", block number string (block size=string)

The ORA-19502 error is raised by Oracle’s backup and recovery infrastructure (most commonly RMAN) when a write operation to a backup piece, image copy, or auxiliary file fails at the block level. It is almost always accompanied by an OS-level error (ORA-27xxx) or a device error that identifies the underlying cause. Because it occurs mid-backup, it can leave backup pieces in an incomplete state, so identifying and resolving the root cause before the next backup window is critical.

Common Causes

1. Backup Destination Full

• Filesystem hosting the backup destination has no remaining space
• Flash Recovery Area has reached DB_RECOVERY_FILE_DEST_SIZE
• ASM diskgroup used for backup pieces is at capacity
• Multiple backup jobs writing simultaneously to the same destination, exhausting space mid-job

2. NFS Mount Issues

• NFS export unmounted or stale NFS handle during a long backup
• NFS server rebooted while backup was in progress
• hard vs soft NFS mount option causing silent failures rather than retries
• NFS write caching issues causing data loss that RMAN detects as a write error

3. ASM Disk Problems

• ASM diskgroup has insufficient free space for the backup piece
• Disk in the ASM diskgroup has failed or is in ERROR state
• ASM rebalance in progress, temporarily reducing write performance and causing timeouts
• Incorrect ASM alias or template configuration for backup files

4. Tape Library Errors

• Tape drive or media manager reported an error to the Media Management Layer (MML)
• Tape cartridge reaching end-of-tape during a backup piece write
• SBT library misconfiguration (SBT_LIBRARY parameter incorrect)
• Media management layer process died while RMAN was writing

5. OS-Level File Descriptor and Permission Issues

• Oracle OS user lacks write permission to the backup destination directory
• Maximum open file descriptors (ulimit -n) reached, preventing new backup piece creation
• Backup piece file already exists from a previous failed job with write protection

Diagnostic Queries

Identify the Failed Backup Job

-- Recent RMAN job history with status
SELECT
  session_key,
  session_recid,
  session_stamp,
  command_id,
  status,
  input_type,
  output_bytes_display,
  TO_CHAR(start_time, 'DD-MON-YYYY HH24:MI:SS')    AS start_time,
  TO_CHAR(end_time,   'DD-MON-YYYY HH24:MI:SS')    AS end_time,
  time_taken_display
FROM v$rman_backup_job_details
WHERE start_time > SYSDATE - 7
ORDER BY start_time DESC
FETCH FIRST 20 ROWS ONLY;

-- Detailed output of failed backup sessions
SELECT
  output
FROM v$rman_output
WHERE session_key = (
  SELECT MAX(session_key)
  FROM v$rman_backup_job_details
  WHERE status = 'FAILED'
)
ORDER BY recid;

Check Flash Recovery Area Utilisation

-- FRA space by file type
SELECT
  file_type,
  ROUND(space_used        / 1024 / 1024 / 1024, 2) AS used_gb,
  ROUND(space_reclaimable / 1024 / 1024 / 1024, 2) AS reclaimable_gb,
  number_of_files
FROM v$flash_recovery_area_usage
ORDER BY space_used DESC;

-- Overall FRA limit vs. usage
SELECT
  name,
  ROUND(space_limit       / 1024 / 1024 / 1024, 2) AS limit_gb,
  ROUND(space_used        / 1024 / 1024 / 1024, 2) AS used_gb,
  ROUND(space_reclaimable / 1024 / 1024 / 1024, 2) AS reclaimable_gb,
  ROUND(100 * space_used  / NULLIF(space_limit, 0), 1) AS pct_used
FROM v$recovery_file_dest;

Inspect Backup Pieces for Incomplete or Unavailable Files

-- Find backup pieces that are not available (failed or deleted)
SELECT
  recid,
  handle,
  status,
  device_type,
  ROUND(bytes / 1024 / 1024 / 1024, 2) AS size_gb,
  TO_CHAR(start_time,      'DD-MON-YYYY HH24:MI:SS') AS start_time,
  TO_CHAR(completion_time, 'DD-MON-YYYY HH24:MI:SS') AS completion_time
FROM v$backup_piece
WHERE status != 'A'
ORDER BY start_time DESC
FETCH FIRST 30 ROWS ONLY;

-- Check for incomplete backup sets
SELECT
  bs.recid                AS bs_recid,
  bs.set_stamp,
  bs.backup_type,
  bs.status,
  bs.pieces,
  bs.device_type,
  TO_CHAR(bs.start_time,      'DD-MON-YYYY HH24:MI:SS') AS start_time,
  TO_CHAR(bs.completion_time, 'DD-MON-YYYY HH24:MI:SS') AS completion_time
FROM v$backup_set bs
WHERE bs.status != 'A'
  AND bs.start_time > SYSDATE - 7
ORDER BY bs.start_time DESC;

Check ASM Diskgroup Status

-- ASM diskgroup space and status (connect to +ASM instance or query from DB)
SELECT
  group_number,
  name,
  state,
  type,
  ROUND(total_mb / 1024, 2)  AS total_gb,
  ROUND(free_mb  / 1024, 2)  AS free_gb,
  ROUND(100 - 100 * free_mb / NULLIF(total_mb, 0), 1) AS pct_used
FROM v$asm_diskgroup
ORDER BY name;

-- ASM disk status
SELECT
  group_number,
  disk_number,
  name,
  path,
  state,
  mode_status,
  total_mb,
  free_mb,
  reads,
  writes,
  read_errs,
  write_errs
FROM v$asm_disk
ORDER BY group_number, disk_number;

Check OS-Level Filesystem Space (via External Table or Alert Log)

-- Query alert log for ORA-19502 and accompanying OS errors
SELECT
  originating_timestamp,
  message_text
FROM v$diag_alert_ext
WHERE message_text LIKE '%ORA-19502%'
   OR message_text LIKE '%ORA-27%'
ORDER BY originating_timestamp DESC
FETCH FIRST 50 ROWS ONLY;

-- Check datafile and backup destinations configured
SELECT
  file#,
  name,
  status,
  ROUND(bytes / 1024 / 1024 / 1024, 2) AS size_gb
FROM v$datafile
ORDER BY file#;

Step-by-Step Resolution

1. Free Space in the Backup Destination

-- Crosscheck and delete expired backup pieces in RMAN:
-- RMAN> CROSSCHECK BACKUP;
-- RMAN> DELETE EXPIRED BACKUP;
-- RMAN> DELETE OBSOLETE RECOVERY WINDOW OF 7 DAYS;
-- RMAN> DELETE ARCHIVELOG ALL COMPLETED BEFORE 'SYSDATE-2' BACKED UP 1 TIMES TO DISK;

-- Increase FRA size if space is the constraint
ALTER SYSTEM SET db_recovery_file_dest_size = 300G SCOPE=BOTH;

-- Check reclaimable space and force RMAN to reclaim it
-- RMAN> DELETE OBSOLETE;

On the OS, check filesystem space:

df -h /backup_dest
# Remove old backup pieces manually only after confirming via RMAN catalog:
# rman target / catalog rman_user/pwd@catdb
# RMAN> LIST BACKUP SUMMARY;

2. Resolve NFS Mount Issues

# Check NFS mount status
mount | grep nfs
df -h /nfs_backup

# Attempt to remount a stale NFS
umount -l /nfs_backup
mount /nfs_backup

# Verify write access
touch /nfs_backup/oracle_write_test && rm /nfs_backup/oracle_write_test

# Review NFS mount options in /etc/fstab
# Recommended options for Oracle backup NFS:
# nfsserver:/backup /nfs_backup nfs rw,hard,intr,timeo=600,rsize=32768,wsize=32768 0 0

-- After remounting, retry the backup:
-- RMAN> BACKUP DATABASE PLUS ARCHIVELOG;

3. Resolve ASM Diskgroup Issues

-- Rebalance ASM diskgroup after adding disks
ALTER DISKGROUP reco ADD DISK '/dev/sdf' NAME RECO_0004;
ALTER DISKGROUP reco REBALANCE POWER 4;

-- Monitor rebalance progress
SELECT
  group_number,
  operation,
  state,
  power,
  actual,
  sofar,
  est_work,
  est_rate,
  est_minutes
FROM v$asm_operation;

-- After diskgroup has space, retry the failed backup
-- RMAN> BACKUP DATABASE;

4. Verify and Clean Up Incomplete Backup Pieces

-- Mark incomplete backup pieces as expired and delete them (RMAN):
-- RMAN> CROSSCHECK BACKUP;
-- RMAN> DELETE EXPIRED BACKUP;

-- Verify remaining backup pieces are accessible
-- RMAN> RESTORE DATABASE VALIDATE;
-- RMAN> RESTORE ARCHIVELOG ALL VALIDATE;

-- Check backup piece integrity
-- RMAN> BACKUP VALIDATE CHECK LOGICAL DATABASE;

5. Correct SBT Library Configuration for Tape

-- Verify SBT parameters (if using tape via MML)
SELECT name, value
FROM v$rman_configuration
WHERE name LIKE '%SBT%'
   OR name LIKE '%PARMS%';

-- In RMAN, reconfigure the SBT channel if the library path is wrong:
-- RMAN> CONFIGURE CHANNEL DEVICE TYPE SBT PARMS 'ENV=(NSR_SERVER=backupserver,NSR_CLIENT=dbserver)';

-- Test the channel
-- RMAN> ALLOCATE CHANNEL c1 DEVICE TYPE SBT PARMS '...'; RELEASE CHANNEL c1;

Prevention Strategies

1. Monitor Backup Destination Space Proactively

-- Create a scheduled job to alert when FRA exceeds 75%
CREATE OR REPLACE PROCEDURE check_fra_space AS
  v_pct_used NUMBER;
BEGIN
  SELECT ROUND(100 * space_used / NULLIF(space_limit, 0), 1)
  INTO v_pct_used
  FROM v$recovery_file_dest;

  IF v_pct_used > 75 THEN
    -- Send alert via UTL_MAIL or DBMS_SCHEDULER notification
    DBMS_OUTPUT.PUT_LINE('WARNING: FRA at ' || v_pct_used || '% capacity');
  END IF;
END;
/

BEGIN
  DBMS_SCHEDULER.CREATE_JOB(
    job_name        => 'FRA_SPACE_MONITOR',
    job_type        => 'STORED_PROCEDURE',
    job_action      => 'check_fra_space',
    repeat_interval => 'FREQ=HOURLY',
    enabled         => TRUE
  );
END;
/

2. Configure RMAN Deletion Policy

-- Automatically delete archived logs applied on standby and backed up
-- RMAN> CONFIGURE ARCHIVELOG DELETION POLICY TO BACKED UP 1 TIMES TO DISK;
-- RMAN> CONFIGURE BACKUP OPTIMIZATION ON;
-- RMAN> CONFIGURE RETENTION POLICY TO RECOVERY WINDOW OF 7 DAYS;

3. RMAN Backup Verification After Every Job

-- Run VALIDATE as part of every backup script
-- RMAN> BACKUP DATABASE PLUS ARCHIVELOG;
-- RMAN> RESTORE DATABASE VALIDATE;

-- Or use VALIDATE CHECK LOGICAL for block-level integrity
-- RMAN> BACKUP VALIDATE CHECK LOGICAL DATABASE ARCHIVELOG ALL;

4. NFS and Storage Best Practices

• Use hard NFS mounts for Oracle backup destinations to ensure retries on transient errors
• Monitor NFS server availability from the Oracle host with a simple cron ping/write test
• For ASM-based backup targets, maintain at least 15% free space in the diskgroup
• Configure RMAN MAXPIECESIZE to create smaller backup pieces, reducing the impact of a single write failure

Diagnostic Scripts

These Oracle Day by Day scripts can help monitor backup and storage health:

• health.sql — Overall database health including FRA status
• db.sql — Database configuration and recovery settings

Related Errors

• ORA-19815 - Flash Recovery Area full warning
• ORA-00257 - Archiver error, connect internal only until freed
• ORA-01578 - Oracle data block corrupted
• ORA-15001 - ASM diskgroup does not exist or is not mounted

Emergency Response

Quick Fixes

1. Free FRA space immediately using RMAN

   rman target /
   # RMAN> DELETE OBSOLETE;
   # RMAN> DELETE ARCHIVELOG ALL COMPLETED BEFORE 'SYSDATE-1';

2. Redirect backup to a different destination

   -- RMAN> RUN {
   --   ALLOCATE CHANNEL c1 DEVICE TYPE DISK FORMAT '/alt_backup/%U';
   --   BACKUP DATABASE;
   -- }

3. Increase FRA size immediately

   ALTER SYSTEM SET db_recovery_file_dest_size = 500G SCOPE=MEMORY;

Post-Resolution Cleanup

-- Cross-check and clean up after the incident
-- RMAN> CROSSCHECK BACKUP;
-- RMAN> DELETE EXPIRED BACKUP;
-- RMAN> LIST FAILURE;
-- RMAN> ADVISE FAILURE;

-- Confirm a clean backup completes after the fix
-- RMAN> BACKUP VALIDATE DATABASE;

-- Review backup job history to confirm success
SELECT
  status,
  input_type,
  output_bytes_display,
  time_taken_display,
  TO_CHAR(start_time, 'DD-MON-YYYY HH24:MI:SS') AS start_time
FROM v$rman_backup_job_details
WHERE start_time > SYSDATE - 2
ORDER BY start_time DESC;