Today, I have data (and code) that confirms the distribution is sorting-algorithm-dependent.  For each sorting algorithm, 1,000,000 instances of the list {0,1,2,3,4} were sorted with a random comparison function and the relative frequencies (rounded to the nearest whole percent) of each number in each position were computed.

The distributions are significantly different among these sorts.  QuickSort appears to provide a uniform distribution.  I believe that this is because QuickSort will only compare a particular pair of elements once, whereas each of the other sorting algorithms may compare a given pair of elements more than once (and with a random comparison function, receive a different result from one time to the next).

Here is the Mathematica notebook I used to generate this data: Randomize by Sorting.nb.  As noted in the file, some of the code for the sorting algorithms was taken from other locations and may be/is subject to their copyrights and/or license terms (I reasonably believe that this use complies with their licenses and/or constitutes fair use.  Also, some algorithms exhibited improper behavior when trying to sort lists with duplicate entries using a normal comparison function as noted in the file, though this should have no effect on the data above.

The second result in a google search for “javascript sort” says:

To randomize the order of the elements within an array, what we need is the body of our sortfunction to return a number that is randomly <0, 0, or >0, irrespective to the relationship between “a” and “b”. The below will do the trick:

//Randomize the order of the array:
var myarray=[25, 8, "George", "John"]
myarray.sort(function() {return 0.5 - Math.random()}) //Array elements now scrambled

This is almost exactly the method of randomization used in the browser ballot javascript.

To test the results of this randomization technique, I applied it 1,000,000 times to the list {0,1,2,3,4} in Mathematica and tabulated the relative frequencies of each number in each position. (Rounded to the nearest whole %).

At a glance, it appears that the distribution is far from uniform.  My quick attempt at re-learning how to use the Χ² test gave a probability less than 1×10^-100000 that this data matched a uniform distribution (if someone can confirm/fix that, please comment).

I used the Mathematica Sort[] command to do the sorting.  I don’t know what algorithm that uses.  It appears that the algorithm used by Javascript’s sort() varies from browser to browser, though the browser ballot would be displayed in IE8 by default.  I suspect that the distribution is highly dependent on the sorting algorithm used, though I cannot readily verify it [edit: I verified it].  Regardless, this seems to be a very poor way to generate a random ordering.