Delete context menu on TFS Source Control explorer will not purge the files and also it need you to get the latest and then delete (more steps), but it does not purge the files.
If you want to permanently delete a TFS branch, use “tf destroy” command on the VS command prompt. Incase if you want to keep the history add “/keephistory” to it.
tf destroy “<branchPath>"
Destroy Command msdn link has full details: http://msdn.microsoft.com/en-us/library/bb386005(v=VS.100).aspx
[It’s been sometime, I purged an branch and I wanted to quickly get the purge comment again and search on Bing and google with keyword permanently delete tfs didn’t bring relevant answer, so thought I’ll post it for permanently delete.]