Web Analytics

How to enable Exadata Write-Back Flash Cache

Yes, this is well-known and the process has been described in Exadata Write-Back Flash Cache - FAQ (Doc ID 1500257.1) but what the note fails to make clear is that you do NOT have to restart cell services anymore hence resync the griddisks!

I had to enable the WBFC many times before and every time I'd restart the cell services, as note suggests. Well, this is not required anymore, starting with 11.2.3.3.1 it is no longer necessary to shut down the cellsrv service on the cells when changing the flash cache mode. This is not big deal if you deploy the Exadata just now but it makes enabling/disabling WBFC for existing systems quicker and much easier.

The best way to do that is to use the script that Oracle has provided - setWBFC.sh. It will do all the work for you - pre-checks and changing the mode, either rolling or non-rolling.

Here are the checks it does for you:

  • Storage cells are valid storage nodes running at least 11.2.3.2.1 or later across all cells.
  • Griddisks status is ONLINE across all cells.
  • No ASM rebalance operations are running.
  • Flash cache state across all cells are "NORMAL".

Enable Write-Back Flash Cache using a ROLLING method

Before you enable WBFC run a precheck to make sure the cells are ready and there are no faults.

./setWBFC.sh -g cell_group -m WriteBack -o rolling -p

At the end of the script which takes less than two minutes to run you'll see message if storage cells passed the prechecks:

All pre-req checks completed:                    [PASSED]
2016-10-10 10:53:03
exa01cel01: flashcache size: 5.82122802734375T
exa01cel02: flashcache size: 5.82122802734375T
exa01cel03: flashcache size: 5.82122802734375T

There are 3 storage cells to process.

Then, once you are ready you run the script to enable the WBFC:

./setWBFC.sh -g cell_group -m WriteBack -o rolling

The script will go through the following steps on each cell, one cell at a time:

  1. Recheck griddisks status to make sure none are OFFLINE
  2. Drop flashcache
  3. Change WBFC flashcachemode to WriteBack
  4. Re-create the flashcache
  5. Verify flashcachemode is in the correct state

On a Quarter Rack it took around four minutes to enable WBFC and you'll this message at the end:

2016-10-10 11:23:24
Setting flash cache to WriteBack completed successfully.

Disable Write-Back Flash Cache using a ROLLING method

Disabling WBFC is not something you do every day but soon or later you might have to do it. I had to do it once for a customer who wanted to go back to WriteThrough because Oracle ACS said this was the default ?!

The steps to disable WBFC are the same as enabling it except that we need to flush all the dirty blocks off the flashcache before we drop it.

Again, run the precheck script to make sure everything looks good:

./setWBFC.sh -g cell_group -m WriteThrough -o rolling -p

if everything looks good then run the script:

./setWBFC.sh -g cell_group -m WriteThrough -o rolling

The script will first FLUSH flashcache across all cells in parallel and wait until the flush is complete!

You can monitor the flush process using the following commands:

dcli -l root -g cell_group cellcli -e "LIST CELLDISK ATTRIBUTES name, flushstatus, flusherror" | grep FD
dcli -l root -g cell_group cellcli -e "list metriccurrent attributes name,metricvalue where name like \'FC_BY_DIRTY.*\' "

The script will then go through the following steps on each cell, one cell at a time:

  1. Recheck griddisks status to make sure none are OFFLINE
  2. Drop flashcache
  3. Change WBFC flashcachemode to WriteThrough
  4. Re-create the flashcache
  5. Verify flashcachemode is in the correct state

The time it takes to flush the cache depends on how dirty blocks you've got in the flashcache and the machine workload. I did two eighth racks and unfortunately, I didn't check the number of dirty blocks but it took 75mins on the first one and 4hrs on the second.