Investigation: Benchmarking

[steve_rogers42]

Hey guys, hows it all been?

So I have been reading through a LOT of the threads on this page with regard to new computers, new installations and new hardware and something has kept me thinking that, maybe, out there, there is a perfect suite or set of tools that would allow us to 'bench' grade or measure the performance of the whole system, as opposed to 'just' fps or how fast your ssd transfers C;/pr0n to D:/pr0n...


So what I'm asking is this, what do you 'use' for benching, or better, what is that game/program/app that grabs your attention when reading performance articles?

A few that come to mind for me are Crysis/Metro2033, Unigine and ATTO

I am preparing to run what we come up with on my system, and if you would like to join me and post progress as i go, your more than welcome to.

I am doing this now as i want to run my own performance index on my computer pre-overclocking, so first i want to find the bench on which to get marks, so that i can then re-bench post OC and see what the differences are. But that will be later, first off, lets just discuss what are good programs for 'benching' and why



Thanks for all your input once again, Let the Science BEGIN!

[SaNE]

What I look at, at the moment:
GPU - Unigine, BF3
MB - Game performance
CPU - Game performance + take a peek at whatever multitasking test the review site has
SSD - AS SSD and/or Crystal Disk Mark
HDD - ATTO and/or Crystal Disk Mark

Of which I usually do a comparison between bit-tech, 3dguru, hexus and anandtech (If I can't find one for the previous sites) for each.
There is World Bench and PC Mark which apparently bench your whole system, I've only seen it on a few reviews though.

[steve_rogers42]

i havent heard of world bench, but pcMark is always generally up there along with 3dMark, synthetic bench's are pretty rad.

sisoft is also pretty interesting to have a look through, although it's more of an information tool than anything... from memory...

Cinebench and quickpar/7zip/winzip runs for file compression etc are generally two i have a look at.

When it comes to real frame per second records with games i tend to be a bit wary, i prefer the sites that will post up min and max frame rates, have a read of THIS is you want to know why...


Cheers SaNE, hopefully we get a few more suggestions...

[SaNE]

i prefer the sites that will post up min and max frame rates

Indeed, average frames are very deceptive.

[Ninja Catfish]

I honestly just use 3DMark.
I benched my SSD one time when shin posted that 'Have you updated your firmware?' thread to make sure my SSD was performing as it should, but that's about it.

[Matty]

It would be cool to have a number of different configs and how they perform. I'll join in, as long as it's not too hard

[Joker]

Some food for thought here! I have used the same benchmark system for decades.

Start with an up to date computer, have a printer attached.

start the timer as you push the power button, wait for the computer to boot, open a word processor, open a document and push print. When the paper starts feeding stop the timer.

Its interesting to note that the best ever performer is windows 2000 on an modern computer at about 30s :p

My current box will do this in 1m45 (dual core w7 on a domain with plenty of startup software including a flash update)
My laptop cheats with about 15s due to sleeping.
A brand new w7 i5 with 2tb drive does this in around 40s
A new HP i5 4730s laptop did this in 45s

Of course there are plenty of variables that effect the test, its not intended to be a epeen benchmark, but more of a real world expected performance test. Windows 7 is one of the better ones and as mentioned w2000 stands out too, vista is one of the worst (surprisingly!) macs dont do that well either, nor any of the mini pc's (hp mini is 10m+ stock). XP used to do over 1m20 on average. My phones android sucks at around 4m though I use the web browser to load a page instead of the printing.

I normally do this on a new pc thats been updated, and has no optional start up software because all the google updaters and the like will slow it down.

[steve_rogers42]

Thats excellent!

Normally boot to google is something i keep tabs on...

Since game performance seems to be one of the major factors, would a spread of games and game styles be worth looking into, if so what would you recommend?

[.J.J.McLowski.]

i use 3dMark and carefully check the differences in numbers on CPU tests and GPU performance and not the total score.

[Long John]

On top of many of the above, I use PlayClaw for in game Benchmark runs and CPU/GPU load and temp monitors and FPS count. Has a nice inbuilt benchmark that you can run and outputs into a spreadsheet. However some games can lag becuase of it.

[steve_rogers42]

Im looking to make this a bit personalized for GoN, so that forum lurkers can use either the existing server's or games that are about, so as such its kinda hard to narrow down...

What im currently thinking is a mix and match of different games across different styles of game as previously mentioned, here is a list of what i am thinking over currently...

FPS:
DX9:
COD / ????

DX10:
Farcry 2 / Crysis

DX11:
BF3 / Metro 2033

OTHER:
Portal 2 / ????

RTS:

SC2 / Hon/Lol / Shogun2 / DoW2

RPG:

DA2 / Fallout: New Vegas / Diablo 3 / ????

Racing:

iracing / Dirt 2 / ????

Synthetic:

3dmark 11 / unigine / pcmark / cinibench

standard:

printing / zipping / ripping / transferring

I am also thinking of Benching 3d performance at different resolutions, so as you can get an idea of what the performance loss is like going between 3d and non-3d modes. Would this be helpful?

Looking to hit up Resolution @ 1080 as its 'the norm' but happy to oblige either way, and also max/ultra vs 'normal' settings with screen shots to show the difference. One thing i miss from review sights is that a lot if missing when they post up fps results and forget to mention how it looks/plays.

Thoughts?

[SaNE]

I am also thinking of Benching 3d performance at different resolutions, so as you can get an idea of what the performance loss is like going between 3d and non-3d modes. Would this be helpful?

3D is a gimmick and I don't know a single person that has actually used it after trying it, it's something different but I don't think it's worth the effort.

Looking to hit up Resolution @ 1080 as its 'the norm' but happy to oblige either way, and also max/ultra vs 'normal' settings with screen shots to show the difference. One thing i miss from review sights is that a lot if missing when they post up fps results and forget to mention how it looks/plays.

You have to decide how much detail you're willing to put into it:
FPS Min / Average / Max
Low / Medium / High Graphics
720p + 1080p + 1440p
Screencaps

It's a lot of work.

[Long John]

Looking to hit up Resolution @ 1080 as its 'the norm' but happy to oblige either way, and also max/ultra vs 'normal' settings with screen shots to show the difference. One thing i miss from review sights is that a lot if missing when they post up fps results and forget to mention how it looks/plays.

Thoughts?

I would try to go mainstream where possible. It's probably best for comparison of A to B etc. You really just need to look at each benchmark respectively and decide on what would be common.

Benchmarks that produce avg FPS/Scores are OK for compare card A to B, but when comparing card A @ xxxmhz they become blurred in results. Personally I go for graphed results like FC2/Metro 2033 as they show better whats happening.

[steve_rogers42]

3D is a gimmick and I don't know a single person that has actually used it after trying it, it's something different but I don't think it's worth the effort.

I actually believe that half the issue with 3D not being used more often, called a gimic etc is due to the performance impact. Similar to multi-screen technology and high resolution gaming, once the technology becomes more improved it will become more available and more used.

Personally i enjoy playing rpg's and 3rd person games with 3d on as its a bit more vibrant and immersive (SWTOR/fallout/skyrim/DA/justcause to name a few) but at the moment i'm using my monitor/tv as a tv so i'm back on my 24" and now its failing on me =P

Not many sites will review games with fps in 3d, i thought this might have been something people might have been interested in from a performance standpoint, but i dont have anything to correlate that if a game runs better in 3d it should run x% better in non-3d scenarios...

plus i have a feeling sli/xfire scaling would run better in 3d benchmarks... but alas i dont have an additional 580 to test...

as for Detail/specs i would be recording min/avg/max then presenting the avg. of each for each run. Runs would be performed @ 1080 resolution on 'high' default as i think it best represents what you want out of the card, and if its playable @ high or not.