Unable to modify or create Protection Groups in DPM 2012 R2 - Error: 31220

"DPM failed to apply the required policies to this protection group since the recovery point limit for this DPM server has been exceeded. (ID: 31220)"

I am trying to add new data sources to our DPM server and continually run into the above message. I get the same error whether I attempt to create a new Protection Group and add the data sources to it... or if I attempt to modify an existing Protection Group to add data sources to it. I can guess that the obvious answer is "You have too many data sources or to many recovery points for the server to handle..." If that's the case, I need to figure out how to tell how many I have. As best as I can figure myself, I have less than 150 data sources and less than 2500 recovery points/snapshot/express-full. If there is a tool or a command that I can run to tell that, I'd be grateful to anyone who can point me in that direction.

I have tried to remove inactive member servers that no longer exist from our Protection Groups... no effect.  I have tried to completely delete the removed server's data from replica... no effect.  We began seeing this issue in 2012 SP1, so I ran through the upgrade to 2012 R2 hoping that it would resolve the issue... no effect.  I am able to remove members from our Protection Groups, but if I try to add an entirely new member or even just a single DB on an already protected member... fails with above message.  I also tried to adjust the retention and backup schedule... fails with above error.

Other than the stock answer from Microsoft that I am truly exceeding some limit, is there any other underlying issue that can cause this to be mis-reported to me?  I hope that SOMEONE has an answer or guidance for me.

May 21st, 2014 4:33pm

You are hitting the 9000 recovery point limit on your DPM. Basically DPM does some calculations to figure out how many recovery points will get created on your DPM server in steady state with the number of data sources that you have already protected + the number you are adding now.

As a rough example, if you have 100 data sources protected and you create 2 recovery points per day and your retention is 30 days, then the number of recovery points in steady state is equal to,

100 * 2 * 30 = 6000.

Use this calculation to see what you can reduce in order to protect more data source son this DPM server. Or better yet, provision another DPM server.

Thanks.


This posting is provided "AS IS" with no warranties, and confers no rights.
Free Windows Admin Tool Kit Click here and download it now
May 21st, 2014 5:46pm

As mentioned in the original post, I am aware of that limit and have done those calculations.  With the information that I am using... my number of recovery points should be less than 2500 at max.  Our retention plans are short and we only do daily express-full backups.  Our retention is 12 days... but I cannot change that retention schedule, EVEN DOWN, without getting the referenced error message. 

Other than a general calculation, are there any tools that can be used to determine EXACTLY how many recovery points exist?  If this is a hard and fast limit that the server will not overcome, why do they not afford SOME way to tell when you are approaching it?

I appreciate the response, but that is a very canned answer.  I am aware of the "stock" Microsoft answer.  My issue is that, as far as I can see, I am in NO WAY hitting the documented limits.  I have been researching this for days and cannot find an answer or even guidance other than "you MUST be hitting the limit"...  That may truly be the case, but if I cannot find how many recovery points I currently have... how can I know how far to adjust down?

May 21st, 2014 5:57pm

There are multiple ways to find out how many recovery points/shadow copies you currently have. vssadmin list shadows will tell you that. Get-DPMRecoveryPoint cmdlet can help you with that too. But that number will not tell you why you are getting this error.

You are getting this error because DPM is determining that you will be hitting this limit in the *future* when you try to protect new data.

This means your calculations should include both existing protection and the new protection that you are trying to do that causes you to see this error.

Thanks.

This posting is provided "AS IS" with no warranties, and confers no rights.

Free Windows Admin Tool Kit Click here and download it now
May 21st, 2014 7:19pm

May be its that you have removed the protection of a member, but it still resides on the disk.
May 21st, 2014 7:24pm

Hi,

This DPM Powershell script should show the provisioning and number of current snapshots for each data source.  Please run it and see what it shows.

  $Script_Name = "Current_Shadow_Copy_Provision"
  $Version     = 2.2


cls
write-host "Script $Script_Name Version $version`n"

$total     = 0
$Totalsnap = 0 
$VSSAdmin = Vssadmin List shadows
Write-host "Found" ($Vssadmin | select-string 'Contents of shadow copy set ID').count "snapshots on this server`n"
$ProtectionGroup = @(get-protectiongroup (&hostname))
"Provisioning Protection Group          # of Snapshots" 
"------------ ------------------------- --------------" 
ForEach ($pg in $ProtectionGroup)
{
	$CurrentSnap      = 0
	$Provisioning     = 0
	$PolicyVolume     = 0
	$PolicyApp        = 0
    $VolumeSnaps      = @()
    $ApplicationSnaps = 
	$ds               = @(Get-DataSource $pg) 
	$VolumeSnaps      = @($ds | ? { $_.type.name -eq 'volume' -or $_.type.name -eq 'Client'} | Sort-Object replicapath | select-object replicapath |get-unique -asstring)
	$VolumeDS         = $VolumeSnaps.count
	$ApplicationSnaps = @($ds | ? { $_.type.name -ne 'volume' -and $_.type.name -ne 'Client'} | Sort-Object replicapath | select-object replicapath |get-unique -asstring)
	$ApplicationDS    = $ApplicationSnaps.count
	$DSTotal          = $VolumeSnaps + $ApplicationSnaps 
	$Policy        = (Get-PolicyObjective $pg -ShortTerm).schedules
	if ($pg.protectionmethod -like '*disk*')
	{
		If ($VolumeDS -ne 0 )
		{
			$PolicyVolume = @(($policy | ? { $_.jobtypestring -eq 'Shadow Copy'}).TimesOfDay).count
		}
		If ($ApplicationDS -ne 0)
		{
			$PolicyApp    = @(($policy | ? { $_.jobtypestring -eq 'Express full'}).TimesOfDay).count
		}
		$Provisioning = $volumeDS * $PolicyVolume * $pg.onsiterecoveryrange.key + $ApplicationDS * $PolicyApp * $pg.onsiterecoveryrange.key
		ForEach ($DSTotal1 in $DSTotal)
		{
			$CurrentSnap = $CurrentSnap + ($VSSAdmin | Select-String $Dstotal1.replicapath.substring(4,44)).count
		}
	}
	$TotalSnap = $Totalsnap + $CurrentSnap

	$output = "{0,12} {1,-25} {2,5}" -f $Provisioning, $pg.friendlyname, $CurrentSnap
	if ($color -eq 'White') { $color = 'cyan' } else { $Color = 'white' }
	write-host $output -f $color
	$Total = $total + $Provisioning

}

$inactive = 0 
$InactiveDS = @()
$InactiveDSTotal  = @(get-datasource (&hostname) -inactive) 
foreach ($ds in $InactiveDSTotal)
{
	If (($InactiveDS | ?  { $_.replicapath -eq $ds.replicapath}) -eq $Null)
	{ 
		$InactiveDS = $InactiveDS + $ds
	}
}

$ActiveDS    = @(get-datasource (&hostname) | ? { $_.UnderActiveProtectionFromPS -eq $true })
ForEach ($ds in $InactiveDS)
{
	if ($ds.iscollocated -eq $true -or $ds.type.name -eq 'volume')
	{
		$CollocatedProtected = @($ActiveDS | ? { $_.ReplicaPath -eq $ds.ReplicaPath}).count
		if ($CollocatedProtected -eq 0)
		{
			[String]$ReplicaPath = $ds.protectableobjectloadpath.values
			$ReplicaPath = $ReplicaPath.Substring($ReplicaPath.IndexOf("vol_"),40)
			$inactive = $inactive + ( $VSSAdmin | Select-String -Pattern $ReplicaPath).count
		}
	}
	else
	{
	    [String]$ReplicaPath = $ds.protectableobjectloadpath.values
		$ReplicaPath = $ReplicaPath.Substring($ReplicaPath.IndexOf("vol_"),40)
        $inactive = $inactive + ( $VSSAdmin | Select-String -Pattern $ReplicaPath).count
		#$inactive = $inactive + @(get-recoverypoint $ds | ? { $_.IsIncremental -eq $False }).count
	}

}
if ($Inactive -gt 0)
{
	$output = "{0,12} Inactive Protection       {1,5}" -f $Inactive, $Inactive
	write-host $output -f magenta
	$total     = $total + $inactive
	$TotalSnap = $TotalSnap + $Inactive
}

"------------ ------------------------- --------------" 
$output = "{0,12}                           {1,5} " -f $total, $TotalSnap
write-host $output -f yellow

Free Windows Admin Tool Kit Click here and download it now
May 21st, 2014 7:40pm

Mike, you're awesome.  That script is AMAZINGLY helpful!!!  Why wasn't something like that baked into the GUI console in DPM?  In my opinion, if Microsoft is going to make limitations based on difficult to calculate metrics... why not give an easily digestible way to see those metrics?

I don't really understand what the "Provisioning" column means, but I can certainly now see why my server is throwing up that error.  My main Protection Group has a Provisioning number of 1308 and 10,285 snapshots... which obviously exceeds the 9000 limit.  The total for all protection groups, including Inactive was over 20,000

What do I need to do to remove some of those snapshots?  I am currently unable to adjust the retention policy to a level that doesn't result in the error.  Am I able to manually remove snapshots?

May 21st, 2014 7:59pm

Hi,

Another DPM PowerShell script - this one allows you to delete older recovery points for a selected datasource without having to modify the PG.

#Author	: Ruud Baars
#Date	: 11/09/2008
#Edited : 11/15/2012 By: Wilson S.
#edited : 11/27/2012 By: Mike J.

# NOTE: Update script to only remove recovery points on Disk. Recovery points removed will be from the oldest one up to the date
#       entered by the user while the script is running
#deletes all recovery points before 'now' on selected data source.
$version="V4.7" 
$ErrorActionPreference = "silentlycontinue"
add-pssnapin sqlservercmdletsnapin100
Add-PSSnapin -Name Microsoft.DataProtectionManager.PowerShell

$MB=1024*1024
$logfile="DPMdeleteRP.LOG"
$wait=10	#seconds
$confirmpreference = "None" 

function Show_help
{
	cls
	$l="=" * 79
	write-host $l -foregroundcolor magenta
	write-host -nonewline "`t<<<" -foregroundcolor white
	write-host -nonewline " DANGEROUS :: MAY DELETE MANY RECOVERY POINTS " -foregroundcolor red
	write-host ">>>" -foregroundcolor white
	write-host $l -foregroundcolor magenta
	write-host "Version: $version" -foregroundcolor cyan
    	write-host "A: User Selects data source to remove recovery points for" -foregroundcolor green
	write-host "B: User enters date / time (using 24hr clock) to Delete recovery points" -foregroundcolor green
	write-host "C: User Confirms deletion after list of recovery points to be deleted is displayed." -foregroundcolor green
	write-host "Appending to log file $logfile`n" -foregroundcolor white
	write-host "User Accepts all responsibilities by entering a data source or just pressing [Enter] " -foregroundcolor white -backgroundcolor blue

}


"**********************************" >> $logfile
"Version $version" >> $logfile
get-date >> $logfile
show_help

$DPMservername=&"hostname"
"Selected DPM server = $DPMservername" >> $logfile
write-host "`nConnnecting to DPM server retrieving data source list...`n" -foregroundcolor green
$pglist = @(Get-ProtectionGroup $DPMservername) # WILSON - Created PGlist as array in case we have a single protection group.
$ds=@()
$tapes=$null
$count = 0
$dscount = 0
foreach ($count in 0..($pglist.count - 1)) 
{
    # write-host $pglist[$count].friendlyname
    $ds += @(get-datasource $pglist[$count]) # WILSON - Created DS as array in case we have a single protection group.
    # write-host $ds
    # write-host $count -foreground yellow 
} 
if ( Get-Datasource $DPMservername -inactive) {$ds += Get-Datasource $DPMservername -inactive}

$i=0
write-host "Index Protection Group     Computer             Path"
write-host "---------------------------------------------------------------------------------"
foreach ($l in $ds) 
{
	"[{0,3}] {1,-20} {2,-20} {3}" -f $i, $l.ProtectionGroupName, $l.psinfo.netbiosname, $l.logicalpath
	$i++
}
$DSname=read-host "`nEnter a data source index from list above - Note co-located datasources on same replica will be effected"

if (!$DSname) 
{
	write-host "No datasource selected `n" -foregroundcolor yellow
	"Aborted on Datasource name" >> $logfile
	exit 0
}
$DSselected=$ds[$DSname]
if (!$DSselected) 
{
	write-host "No datasource selected `n" -foregroundcolor yellow
	"Aborted on Datasource name" >> $logfile
	exit 0
}
$rp=get-recoverypoint $DS[$dsname]
$rp
# $DoTape=read-host "`nDo you want to remove when recovery points are on tape ? [y/N]"
# "Remove tape recovery point = $DoTape" >> $logfile

write-host "`nCollecting recoverypoint information for datasource $DSselected.name" -foregroundcolor green
if ($DSselected.ShadowCopyUsedspace -gt 0)
{
    while ($DSSelected.TotalRecoveryPoints -eq 0)
    { # "still 0"
    }

	#this is on disk
	$oldShadowUsage=[math]::round($DSselected.ShadowCopyUsedspace/$MB,1)
	$line=("Total recoverypoint usage {0} MB on DISK in {1} recovery points" -f $oldShadowUsage ,$DSselected.TotalRecoveryPoints  ) 
	$line >> $logfile
	write-host $line`n -foregroundcolor white
}

#this is on tape
#$trptot=0
#$tp= Get-RecoveryPoint($dsselected) | where {($_.Datalocation -eq "Media")}
#foreach ($trp in $tp) {$trptot += $trp.size }
#if ($trptot -gt 0 )
#{
#	$line=("Total recoverypoint usage {0} MB on TAPE in {1} recovery points" -f ($trptot/$MB) ,$DSselected.TotalRecoveryPoints  ) 
#	$line >> $logfile
#	write-host $line`n -foregroundcolor white		
#}

[datetime]$afterdate="1/1/1980"
#$answer=read-host "`nDo you want to delete recovery points from the beginning [Y/n]" 
#if ($answer -eq "n" )
#{
#	[datetime]$afterdate=read-host "Delete recovery points AFTER date [MM/DD/YYYY hh:mm]"
#}
[datetime]$enddate=read-host "Delete ALL Disk based recovery points BEFORE and Including date/time entered [MM/DD/YYYY hh:mm]" 
"Deleting recovery points until $enddate" >>$logfile
write-host "Deleting recovery points until and $enddate" -foregroundcolor yellow
$rp=get-recoverypoint $DSselected

if ($DoTape -ne "y" )
{
	$RPselected=$rp | where {($_.representedpointintime -le $enddate) -and ($_.Isincremental -eq $FALSE) -and ($_.DataLocation -eq "Disk")}
}
else
{
	$RPselected=$rp | where {($_.representedpointintime -le $enddate) -and ($_.Isincremental -eq $FALSE)}
}

if (!$RPselected) 
{
	write-host "No recovery points found!" -foregroundcolor yellow
	"No recovery points found, aborting...!" >> $logfile
	exit 0
}
$reselect = $enddate
$adjustflag = $false
foreach ($onerp in $RPselected)
{
	$rtime=[string]$onerp.representedpointintime
	$rsize=[math]::round(($onerp.size/$MB),1)
	$line= "Found {0}, RP size= {1} MB (If 0 MB, co-located datasource cannot be computed), Incremental={2} "-f $rtime, $rsize,$onerp.Isincremental
        $line >> $logfile
	write-host "$line" -foregroundcolor yellow
	#
	#Get dependent rp's for data source
	#
	$allRPtbd=$DSselected.GetAllRecoveryPointsToBeDeleted($onerp)
	foreach ($oneDrp in $allRPtbd)
	{
		if ($oneDrp.IsIncremental -eq $FALSE) {continue}
		$rtime=[string]$oneDrp.representedpointintime
		$rsize=[math]::round(($oneDrp.size/$MB),1)
		$line= ("`t...is dependancy for {0} size {1} `tIncremental={2}" -f $rtime, $rsize, $oneDrp.Isincremental) 
		$line >> $logfile
		if ($oneDrp.representedpointintime -ge $enddate)
		{
			#stick to latest full ($oneDrp = dependents, $onerp = full)
			$adjustflag = $true
			$reselect = $onerp.representedpointintime
			"<< Dependents newer than BEFORE date >>>" >> $logfile
			Write-Host -nonewline "`t <<< later than BEFORE date >>>" -foregroundcolor white -backgroundcolor red
			write-host "$line" -foregroundcolor yellow
		}
		else
		{
			#Ok, include current latest incremental
			$reselect =  $oneDrp.representedpointintime
        	write-host "$line" -foregroundcolor yellow
		}
	}
}
if ($reselect -lt $oneDrp.representedpointintime) 
{
	#we adjusted further backward than latest incremental within selection
        $reselect =  $rtime 
	$line =  "Adjusted BEFORE date to be $reselect to include dependents to $enddate"
	$line >> $logfile
	Write-Host $line -foregroundcolor white -backgroundcolor blue
}
$line="`n<<< SECOND TO LAST CHANCE TO ABORT - ONE MORE PROMPT TO CONFIRM. >>>"
write-host $line  -foregroundcolor white -backgroundcolor blue
$line >> $logfile
$line="Above recovery points within adjusted range will be permanently deleted !!!"
write-host $line -foregroundcolor red
$line >> $logfile
$line="These RP's include dependent recovery points and may contain co-located datasource(s)"
write-host $line -foregroundcolor red 
$line >> $logfile
$line="Data source activity = " + $DSselected.Activity
$line >> $logfile
write-host $line -foregroundcolor white
$DoDelete=""
while (($DoDelete -ne "N" ) -and ($DoDelete -ne "Y"))
{
	$line="Continue with deletion (must answer) Y/N? "
	write-host $line -foregroundcolor white
	$DoDelete=read-host
	$line = $line + $DoDelete
	$line >> $logfile
}

if (!$DSselected.Activity -eq "Idle") 
{
	$line="Data source not idle, do you want to wait Y/N ? "
	write-host $line -foregroundcolor yellow
	$Y=read-host
	$line = $line + $Y
	$line >> $logfile
	if ($Y -ieq "Y") 
	{
		Write-Host "Waiting for data source to become idle..."  -foregroundcolor green
		while ($DSselected.Activity -ne "Idle")
		{
			("Waiting {0} seconds" -f $wait) >>$logfile
			Write-Host -NoNewline "..." -ForegroundColor blue
			start-sleep -s $wait
		}
	}
}

if ($DoDelete -eq "Y")
{
   foreach ($onerp in $RPselected)
	{
		#reselect is adjusted to safe range relative to what was requested
		#--- if adjustflag not set then all up to including else only older because we must keep the full
		if ((($onerp.representedpointintime -le $reselect) -and ($adjustflag -eq $false)) -or ($onerp.representedpointintime -lt $reselect))
		{
			$rtime=[string]$onerp.representedpointintime
			write-host `n$line -foregroundcolor red
			$line >>$logfile
			if (($onerp ) -and ($onerp.IsIncremental -eq $FALSE)) { remove-recoverypoint -RecoveryPoint $onerp -confirm:$True} # >> $logfile}
			$line =("---`nDeleting recoverypoint -> " + $rtime)
			$line >>$logfile
		}
	}
}
"All Done!" >> $logfile
write-host "`nAll Done!`n`n" -foregroundcolor white
$line="Do you want to View DPMdeleteRP.LOG file Y/N ? "
	write-host $line -foregroundcolor white
	$Y=read-host
	$line = $line + $Y
	$line >> $logfile
	if ($Y -ieq "Y") 
	{		
	Notepad DPMdeleteRP.LOG
	}
Free Windows Admin Tool Kit Click here and download it now
May 21st, 2014 8:11pm

Mike, I need to thank you for the first script that you provided... that is exactly what the DPM GUI console is missing. 

I was able to track down where my rogue recovery points were and take care of them.  The second script never really worked for me, but that's a moot point because I was able to resolve the issues manually with the information gained from the first script.  After days of trying to find a way to dig us out of the hole we'd gotten into with DPM, that script was a life saver!

May 22nd, 2014 6:04pm

This script returns a figure less than 4000 but we're still encountering the error, even when creating a recovery group including 1 server, retention period 24 hours and 1 backup/day. This would only use 1 recovery point in my opinion, certainly not 5000.

any ideas why this error keeps popping up?


Free Windows Admin Tool Kit Click here and download it now
November 25th, 2014 12:46pm

I've run the script also, and I have 3364 snapshots on primary dpm server, 1183 snapshots on my secondary DPM server, and still get the error with SQL auto protection on the secondary and autoprotectinstances.ps1 fails.

Any ideas?

November 25th, 2014 1:15pm

I have just removed the SQL autoprotected datasource from the protection group (without removing the content) and added it again. This has worked whereas trying to resolve the auto protection generated the errors about too many snapshots. Odd behaviour, but worked around. Shame the error is so inaccurate then!
Free Windows Admin Tool Kit Click here and download it now
November 25th, 2014 1:40pm

When a storage pool loses access to a volume DPM marks the volume as "missing" in the tbl_RM_ShadowCopy table.  DPM does not know if the volume will come back online and be reconnected so it skips GC.
If the volume never comes back these items never get removed from the DPMDB.  The issues occurs because the DPMLimitsValidator.cs counts these entries against the global 9000 VSS shadow copy count.  Once it hits 9000 DPM thinks all shadow copies are taken and stops additional datasources from being added.

March 26th, 2015 2:25am

This topic is archived. No further replies will be accepted.

Other recent topics Other recent topics