…from renowned journalist David Pogue, who was wondering why Pono doesn’t sound any different…
“I found 15 volunteers, ages 17 through 55. Each subject put on nice headphones — Sony MDR 7506 — and listened to three songs of different styles (“Saturday in the Park” by Chicago, “Raised on Robbery” by Joni Mitchell, and “There’s a World” by Mr. Pono himself, Neil Young). I bought these songs twice: once from the Pono store, in high resolution, and once from the iTunes store.
Each subject then listened to the same songs again, using standard Apple earbuds.
I connected both the Pono Player and an iPhone to an A/B switch; I instructed my test subjects to flip back and forth between the two at will.”
…and the results?
“Whether wearing earbuds or expensive headphones, my test subjects usually thought that the iPhone playback sounded better than the Pono Player.”
Check out the exhaustive review here.
certainly not snakeoil selling, just consumers who dont understand audio thinking they have a right to say anything about it or that their lame opinion matters at all…
Sure, it’s snakeoil:
All blindtests prove that nobody can tell the difference between modern iTunes downloads and 16/44.1 or 24/96.
Want great sound? Buy great headphones.
“nice headphones — Sony MDR 7506”
hahahahahahahahahahahahahahahahahahahahaha
hahahahahahah. Lmao
Hahahaha
They are a perfectly acceptable, and smart, choice when it comes to consumer testing. Would you really suggest that the results would be more/less valid using headphones costing more than consumers would normally pay?
Yeah, the 7506 is a good closed back headphone. But, they are certainly not “flat” or even “good” sounding. I use mine all the time to solo inputs in a live environment, the mid range boost is great for competing with the noisy live environment. But, for pure listening, I go for my AKG’ open backs (75.00) or my Sennheiser’s. Although my guess is that the compression of the mp3’s and the extra volume boost of those files is why folks picked them. If the files were not volume matched, and if the test was not “double blind” it’s meaningless.
I agree! – Not enough detail, insight, or transparency in this study.
Well, yes…of course.
Do you think the test should be accomplished using earbuds?
😉
he he he he he he he he!
This was actually a super bogus test. Provided that the itunes downloads were “mastered for itunes,” they SHOULD sound better through the earbuds because something like an eq job that compensates for the earbids’ deficiencies. So right off the bat, the results should be split 50/50. Beyond that, the earvuds/headphones are run through some cheap looking device that allows users to swap from device to device. In an analog path, your signal is only az good as your weakest link, so for all we know thos device could be undermining not only the upgraded circuitry of the pono player, but potentially also the fidelity of the source file.
It’s super hip hate on Pono right now. If youre gonna do it, at least do a proper a/b
This. Critical path, headphones, the fact that more often than not MfiT is mastered louder and more compressed…
Would seem that the Pono/FLAC source would reveal the shortcomings of the earbuds, whereas the iPod/AAC would cover them up, making the latter more likely to be chosen.
That’s not something you can ask the average consumer to recognize.
this writer is a moron, the RCA Splitter from RadioShack is equalizing the crappyness of each signal. I think Pono is retarded but this is just a cheap shot and unfair defamation.
Some commenters have expressed concern about the Radio Shack A/B switch in my test, and the RCA cables connected to it. This is what my audio engineering friends say in response:
“The switch is just a copper crossconnect. There is literally nothing that can go wrong with it . As I like to say when advising people against buying expensive ‘high-end’ audio cables: copper is copper.”
As for the cables: “As an audio/video engineer of 30 years, I can assure you the RCA cable introduces no distortion, assuming the cable is correctly built. Even the cheapie Chinese cables these days are perfectly equal to the most expensive Monster Oxygen-free $100 gold-plated scam cables. In fact, RCA makes a much better connection that 3.5mm plug, since it has many times the surface area and clamping pressure.
The highest AAC audio bandwidth of 192kHz is easily carried by the 10MHz capacity of RCA cables.”
Hope this helps!
@David Pogue
I appreciate your willingness to discuss the test.
To me, it makes sense that more would prefer the earbud songs through the phone, given that this is a native environment. It makes sense that the earbuds were designed around the shortcomings of compressed audio & vice versa. For the Sony headphones, would you say that people who were confident about their choice, even if they were wrong, were identifying a sonic difference, even if they didn’t know which was the “right” difference? Could it be that they were choosing the phone as superior because of sonic signatures or artifacts that they were used to (consciously or not), not knowing which was the version intended by the artist, or which version most closely reflects the original live performance? Should you put the onus on the average listener to know what a realistic cymbal crash or acoustic guitar strum sounds like, versus one that might be distorted or otherwise colorized in some way? (Certainly, the average consumer has heard many more compressed, colored cymbal crashes than real, live, present cymbals, & may associate the sonic artifacts with the instrument itself.) Additionally, at higher volumes & through nicer headphones, would you agree that the difference identified in ~75% of cases would be exaggerated?
There seems to be a bias present in the test, as if the intent was to steer the average consumer away from the Pono device. Would it be more valuable to the consumer if you, instead, had sought out what components would be required to produce a more clearly differentiated sound, regardless of the user’s preference?
To me, this is a lot like the EQ feature on home stereos or computer software. EQs can be very useful when compensating for a specific speaker or a room’s sonic signature. But in the hands of the average consumer, they generally crank up the highs & the lows & maybe the mids to give the illusion of dynamics, effectively destroying the subtleties & perhaps even the basic integrity of the mix. More often than not, the average consumer is going to use the EQ, if they interact with it at all, to act against the wishes of the artist or mixer. So it’s not very surprising that the average consumer may choose an iTunes file over a flat, non-distorted HD file. What’s important to me is that they noticed that there is a difference (and they seemed to ~75% of the time), even through sub-$100 headphones. To me, that suggests that the user can, given enough exposure, learn to recognize & subsequently appreciate a flat, natural sounding audio recording.
It would be interesting to give a user this test, then have them listen to the Pono player exclusively through nice headphones (nicer than the ones used here) for a week or two, & then have them take the test again & compare the results.
Also, are you considering yourself one of the 15 volunteers? Or were your results excluded from this data?
One final note, it should be said that some portion of the population, whatever that is, assuming there is a difference in fidelity between an iTunes download & a Pono full res download (I’m assuming there is, based on your data), some portion of consumers will not be able to appreciate it. This could be because they physically lack the ability to hear the subtleties, or because they lack the musical IQ to be able to distinguish individual instruments. Some people only register the vocal melody or very obvious instrumental hooks. And so it’s ok that not everyone, even in perfect listening conditions, can hear the difference. That doesn’t mean it doesn’t exist. Perhaps a good baseline would be to compare an iTunes download & a Pono download that are clearly two different mixes, & if the user can’t tell the difference, then it’s hard to blame Pono. (For my own personal test I used the CD version & SACD version of REM’s In Time compilation, comparing an mp3 ripped from CD & endcoded with Lame at v0 settings to a 24-bit/96khz FLAC SACD rip.)
For what it’s worth, I’ve never listened to hi-res FLAC through a Pono player.
I suggest an A/B test with informed listeners (e.g., people who listen to music as a profession such as recording engineers, musicians (thinking more Yo-Yo Ma than Deaf Leopard (that’s meant to be a joke)), conductors, etc.) or audio equipment reviewers, or other categories of listeners who have an idea of what to listen for. I’m a long-retired recording engineer who can in an instant tell the difference between my Pono with AIF files and my iPhone 6+ with iTunes and so-called Apple Lossless files. And I can hear the difference whether Bose QC15s or Grado GS1000. Sounds “Good” is completely subjective (I have a friend who listens to 60’s hits on a single Advent speaker being pushed too hard to get that ‘good’ sound he loves about 60’s music). Sounds like what was likely intended by those who made the recording can be a bit less so, but only with trained ears. Btw, I haven’t tried ear-buds, but if you are an ear-bud listener, than why bother with the Pono?
Quote “The switch is just a copper crossconnect. There is literally nothing that can go wrong with it . As I like to say when advising people against buying expensive ‘high-end’ audio cables: copper is copper.”
.
As an electronic engineer, former theater audio guy, and a DJ
I will easily DITTO that comment..
You are 100% correct, Mr Pogue. The A-B switch would have been the least distorting component in the entire configuration. And as you say, even the cheapest RCA interconnects are more than capable of carrying a basic 192kHz sampled signal.
The reality is the CD format of 16-bit/44.1kHz is more than enough for every human, without exception. There are no humans who have hearing better than 96dB SPL (actually 120dB with dithering) and 20kHz bandwidth. None. Nobody. Not one. The CD format was intentionally over-engineered and 30 years later the limits of human biology has not changed. There are no human ears that can hear the frequencies that 192kHz sampling provides, and no human ears sensitive enough to hear 24-bits dynamic range. 100+ years of rigorous scientific testing has proven this beyond any doubt.
Naysayers should read xiph.org and hydrogenaudio. The maths and physics is beyond doubt. Controlled testing has also proven that nobody, not even golden-eared professionals, could hear any difference between a properly mastered 16/44.1 recording and a properly mastered 24/192 recording.
I’m an electronics engineer and an audio professional with 20+ years experience. Your A-B test was perfectly fine and used appropriate equipment for the intended audience (e.g. headphones that the average pro-sumer is likely to own). I agree with you 100%. This Pono product is well-intentioned but offers nothing tangible.
> Anonymous finally says “i know nothing,”
You got that right. Shame it took you nearly 1000 words to get to the point.
16-bit / 44.1kHz is all you need to encode audio for human ears. Here’s a very informative article at xiph.org aimed at anybody with an open mind.
http://xiph.org/
You don’t need to be an expert to understand the xiph.org article. So take the time to educate yourself and improve yourself at the same time.
16-bit / 44.1kHz is all you need to encode audio for human ears. Here’s a very informative article at xiph.org aimed at anybody with an open mind.
http://xiph.org/
You don’t need to be an expert to understand the xiph.org article. So take the time to educate yourself and improve yourself at the same time.
I agree completely with the gist of your point, save for the fact that there are individuals that have been shown to hear above 20k. Only a few kHz, perhaps, but worth mentioning. And no, this doesn’t make 16 bit/44.1 kHz inadequate.
David, thank you so much for doing this!
as a musician and performer that makes their living off of recorded and live music, i wouldn’t be caught dead using a throughput made by radioshack. you will inevitably introduce impurities based on the cheapness of the radioshack switch. i’ve never used anything from RS that wasn’t a complete staticy piece of trash. Ask Marissa Meyer and Jack Ma for some of that AliBaba money from yahoo before you write your next story using radioshack garbage.
I don’t understand why you used two different sets of headphones. If you were testing the Ponos player and compression vs. iTunes/iPod, why use two different sets of ‘speakers/headphones/earbuds’ or were the apple earbuds intended to be part of the ‘study’?
Aside from speakers (read headphones) the only other items that influence the sound are your power and preamps (those would be your players) Then comes the recording (compression technique). Switching between headphones puts in a third variable, which doesn’t count since people swap headphones often.
I omitted EQ, but hopefully it was flat on the iPod and on the Ponos (if an EQ exists on Ponos. I don’t know if they have enhancement features)
I won’t disagree with your results, in the past, I’ve preferred the iPod over other devices, now I’m using one which is more on par, I’m just questioning your scientific method.
If Ponos doesn’t make headphones, and you were trying to compare what’s available ‘stock’, the least you should do is use the Apple earbuds when playing the Ponos.
I don’t disagree. I have high end audio equipment, and even a $150 Sony Receiver, will get you almost where a $2100 Amp and preamp combo.Its all snake oil. For haters, just do an A/B test ourself and report back. Empirical evidence is the best way to test
Why not use a decent set of headphones like the new BOSE QC 25? I own a set and find that it reveals all. I’d be really interested in your results then.
Ps. ( I too am an experienced musician with 30 years performing, writing, recording etc.)
The idea behind testing is to get feedback based upon normal use. The Sony headphones provide a real life scenario. By using a high-end set, the test is tainted.
The $100 price point for headphones was arbitrarily chosen. Pono themselves recommend headphones in the $200-300 price range.
The test probably should’ve been conducted on some Beyerdynamic, Sennheiser, or Grado headphones in that price range, or something comparable. Something 32-40 ohm, not the 63 ohm Sonys.
Pono indeed recommends super-expensive phones. Just wanted to +1 that.
It is quite interesting to bring that topic up, and I’ll use this opportunity to share a personal experience around Pono testing on random people (not audiophile / HD audio evangelists).
I attended an event (FMF in Barcelona last September) where there were a couple of Pono reps (and some of their tech partners).
At coffee break the group of people I was talking with was offered the option to listen to various songs loaded on a Pono prototype one of the guys had brought to the event.
The “test” was to listen to Led Zeppelin songs (I presume these were the remastered versions) on the Pono device.
Interestingly, they were using DT770 Pro headphones (headphones widely used for recording sessions in studio) plugged into the Pono.
Of course the audience was stunned by the “great” and “amazing” sound quality of Pono…
If you ever used such headphones, you certainly know that any audio source with a decent quality will sound great, you’ll have dynamics, presence…etc, these are (mid-range) professional headphones, nothing surprising with this.
So I asked the guys if it was possible to listen to the same remastered files on an iPhone, using the same headphones + suggested to run another test with standard earbuds and compare Pono and iPhone.
Embarrassed silence / changed topic.
As I’ve mentioned before, let’s ask Paul Young to start conversion of Radio & streaming to simple music store.
Someone hose to do it, or labels will continue the trans of great licensing deals with Eks and the tubes.
Music doesn’t have to be drained into the TUBES!!! Bright future is just around the corner.
This spam is really getting out of control.
I am real with primitive solution. You are a lot of intelligent talk on irrelevant, at least for music industry, issues.
Indeed.
This is just as bogus as the blind wine taste-tests they use in my industry. Here’s how this works – you choose 15 random people and you get just that… 15 random opinions. Now I dare you to take -5 audio professionals that work in a studio, let them choose their own headphones, and do this test again. I too think Pono is likely a full of shit idea. But until you choose to test it with people who KNOW what to listen for, this kind of test is bullshit. Fool a pro and I’ll listen to your study results any day. Fool a random guy and you’re just showing that Companies have done a great job adapting their crap headphones And production styles to each-other. Lots of listeners these days think the unnatural sounds of compressed music are “good.” So why use those people as your test subjects??? It’s just like the wine tests – a bunch of randoms ALWAYS like the shittier wine – why? Because it’s easy to drink grape juic with alcohol. Real wine is not always so “easy” and usually wine professionals will pick the differences out in about 2 seconds while the random consumers have no clue what they’re tasting.
And David, please, do repeat this study with a group of audio professionals and some actually nice headphones. And maybe take away the radio shack switch just to make everyone else happy too…? If the results are the same then we can discuss 🙂
Guys – yes audio pros will tell the difference. The point is that “everyman” cannot or does not care. I assume Ponos, as a consumer product, is not just looking to sell to a very niche market so the study perfectly valid.
There is no mass market need for a Ponos.
Yes, however if they cannot claim that even audio pros can tell the quality difference in a blind study, then the company fails entirely and has nothing to market on… The data already suggests little to no difference when compared to high quality digital compressed files.
15 volunteers? That’s it?
The average listener also prefers Skrillex to Bach or Miles.
+ O I
Thank you to Mr. Pogue for the effort, and a double-blind test is a good idea. There may indeed be a negligible difference between the Pono and iPod. However, this test is inadequate.
First, Pono is meant to be high fidelity. So let’s test on a high fidelity system, whether high-end headphones or sound system with speakers.
Second, were volumes compensated, including taking into account the extra compression possibly applied on the iTunes versions? If even a slight difference in volume, listeners will tend to “prefer” the louder version.
Third, let’s at least include some music which at least potentially reveals the advantages of high-fidelity. Classical, jazz, more dynamic rock.
Hopefully someone will perform a more extensive and thorough double-blind test soon.
exactly
Useless waste off development time.
well. higher quality audio actually makes sense.
but testing random people doesnt. for most people listening through your laptop speakers is just fine
Yep, those are the songs I would download and test…puhleeeeze
Supporters of Pono and so called “hi-def” audio are just magical thinkers. No use whatsoever in convincing them that they actually just prefer the placebo.
@Asdf, you say “Supporters of Pono and so called ‘hi-def’ audio are just magical thinkers. No use whatsoever in convincing them that they actually just prefer the placebo.” Who are you talking to here? I don’t know anyone who knows a thing abou hi def audio that would think a Pono player is any better than other options already on the market. Furthermore, don’t group all hi def audio listeners into one group. Many of us don’t fall prey to expensive copper or snake oil sales. But there’s still major differences between audio options that are far from ridiculous and don’t deserve your mocking attitude. Not all of us are wealthy jerks that fall prey to gimmicks, and any real hi def audio listener can blind listen to differences… Many times proving that less expensive and “regular options” are amazing. But don’t start generalizing because some of us can hear the differences between total crap speakers or headphones and good stuff. Not based on price, based on sound. Pono doesn’t add up in the technical data category And therefore assume it won’t add up in the blind listening category. But let the pros decide… IE people who get paid to listen to, mix, and produce music. Critical listening is an entire category of music studies. Don’t deny.
Reply
I bought a number of Apple iPods over the years, but stopped “upgrading” because it was a constant hit and miss.
The 1st-gen mini and the 30GB iPod Photo made music that wrapped its arms around me like a poppy-fueled dream. My cold dead hands, yadda yadda yadda. The nano (1st ed, IIRC, black and flat), and the 2nd-gen mini (which supposedly corrected the short battery life problem of the 1st-gen mini by using a different chipset which, you know, would ALTER THE SOUND!?!?), eh — annoying. Hated listening to them.
Just my $.02
Just wanted to add that I discovered the chipset switch only AFTER trying to research why it might be that I loved my Mini I and hated my Mini II.
Proper quality music includes all the sounds recorded and is likely to make people FEEL better as it will have all its SOUL.
The average person cannot ‘hear’ the difference as their ears are not trained. After I did a mixing course, including listening training, it is now very obvious to me.
So, its very hard to measure people’s preference responses to music when a) they are not trained to hear properly and b) Any extra compression or loudness will increase the bias towards that track. c) the gear they are listening on doesn’t do it justice. You are hardly likely to buy a Pono and use apple earbuds.
The best A/B test is iPhone and earbuds v Pono and good headphones. Then ask the question – Would you pay the extra dollars for the Pono and good headphones?
SOUL
Exactly what frequency range does the “soul” of the music lie?
What a fascinating thread! I think this test, while not scientifically perfect, still provides some interesting anecdotal information. I would be really interested in hearing from any of the commenters that currently use anything OTHER than iTunes for managing their music collection about what they use and why. I would really like to get iTunes out of my life permanently, for a number of reasons. It doesn’t look like Pono is a very viable option for now, so I don’t know where to turn. Does anything else do most of what iTunes does at decent fidelity? Or have they successfully made themselves the only real player in the game? Any recommendations?
Neil Young should sit for the A-B test himself and we’ll be done with this.
Not surprising. I would of loved seeing something more based on hardware than track source. ie, can the average person tell the difference of songs both purchased on itunes and hdtracks on the latest ipod/iphone compared to the pono or other “portable audiophile” players.
I see no reason to A-B myself to death. If I have to choose between 256kbps AAC and 44.1/16 “lossless” or higher, I’m gonna take the latter. Mastering/generation details aside, the former is sure to waste my time. The latter, no more than buying a CD like I did in the 80s and 90s. Yes, I will,at the end of the day, have bought something crappily mastered along the line. But that’s life.
HW: AFAIK, no Apple hardware yet plays hi-res (192/24) files. And I haz dem.
With one exception, Pono is the least expensive hi-res capable PMP out there. Go above Pono, and you’re hitting four digits. Not going there, personally. And without a hi-res capable PMP, I’m tethered to my home system.
Most Mac computers will output 24/96, you just have to set it up in Audio MIDI setup. Alternatively, you can HDMI or Thunderbolt-to-HDMI into any modern receiver, which should be the full resolution digital signal. (This is also how you playback 5.1 from a mac.)
I have hi-res desktop playback.
I don’t have anything for hi-res on-the-go, which is partially the point of pono.
Thank you for the test, i find it very interesting.
However i second the idea of repeating the same test with professional audio guys and professional headphones. Simply because random people is trained to recognize mp3 as “good” and often they don’t even perceive the difference between a 128kbps track and a 256kbps.
Maybe if higher quality audio will become the standard also random people will be able to recognize it. 😉
I suggest an A/B test with informed listeners (e.g., people who listen to music as a profession such as recording engineers, musicians (thinking more Yo-Yo Ma than Deaf Leopard (that’s meant to be a joke)), conductors, etc.) or audio equipment reviewers, or other categories of listeners who have an idea of what to listen for. I’m a long-retired recording engineer who can in an instant tell the difference between my Pono with AIF files and my iPhone 6+ with iTunes and so-called Apple Lossless files. And I can hear the difference whether Bose QC15s or Grado GS1000. Sounds “Good” is completely subjective (I have a friend who listens to 60’s hits on a single Advent speaker being pushed too hard to get that ‘good’ sound he loves about 60’s music). Sounds like what was likely intended by those who made the recording can be a bit less so, but only with trained ears. Btw, I haven’t tried ear-buds, but if you are an ear-bud listener, than why bother with the Pono?
stupid peoples mistakes volume with quality, hence de sucess of loudness mixing and lossy formats, this test is a complete bluff.
I really don’t care about 15 random people with sony headphones, or any other for that matter. The point is THAT I CAN tell the difference. I’ve used ipods for years, most currently the 160gb classic. I’ve always been frustrated at the inconsistency of some music sounding better than other. I bought a Pono player about a month ago and can say without any shadow of a doubt my music downloaded in 24/96 blows away any music on my ipod. I use iem’s, very nice/expensive ones at that. The Pono isn’t for the curious, it’s for the obsessed. If you know what you want to hear, you’ll tell a difference. It’s not magic, it’s not voodoo and it’s not bullshit. I don’t care what your science books or million dollar education say. I let my ears decide, and they did. If I couldn’t tell the difference I was prepared to send it back, but the longer I used it and got it broke in, the fuller the music became. If you aren’t serious, obsessively serious about your music, don’t waste your cash. But if you know what you’re listening for and are willing to drop $400.00 on it, I strongly urge you to try it before you post negative comments without ever trying it. And yes, good headphones/earbuds DO make a difference.
Yep, ya dead right, I can tell the difference very easily with high res – after listening to dvd-audio, blu ray audio discs & lp’s on my system, 90% of my cds sound like crap & it erks me to play them these days. It’s like going back to VB after discovering good craft brews. Every single one of the visitors (around 40 i’d say) that have listened in my lounge to the dvd-a of Steely Dan’s Gaucho, or the blu ray audio of Tom Petty’s Hypnotic Eye has smiled & gone “wow, sounds amazing”. My system is a reasonable but moderate one.
Good sound engineers bust their arses to capture the best possible sound for a recording & some idiots want to dumb it down & tell you you can’t tell the difference! Get your hands off it…..
A similar test was done by Tom’s Hardware a while back, using two audiophiles. Again, they could not reliably tell between different equipment.
http://www.tomshardware.com/reviews/high-end-pc-audio,3733-19.html