1. How many trials can the AI do? If it scores on 1/100 tests is that sufficient? How do you guarantee a fresh unseen test?

2. What is AI? eg can it use wolframalpha, the internet, etc? Does it need to be an LLM? Can it be GPT4 with plugins? etc

also see this market:

@PatrickDelaney tangentially related

I think that the SAT verbal section is something of a specific learned skill. You sort of learn to intuit which answer is "most defensible" rather than "correct." Getting an 800 seems hard without some kind of fine-tuning or prompt engineering. Of course it could happen by chance as a perfect score allows some incorrect answers.

@salix it's entirely luck above 6 or 700, you can do the distribution and see its luck

These comments

For math problems yes?

For the bullshit verbal section? No. Not even for humans more than once.

@MarkIngraham for both sections for a combined 1600 perfect score

@MarkIngraham it's definitely possible to get perfect scores on the verbal section consistently

@VictorLi that would be the Turing test, you would never use an ai in a stupid way if it was that powerful

@MarkIngraham the english section is nowhere close to a Turing test in terms of difficulty, and GPT-4 can already reach over 700 for it. The only question is whether or not they can polish the AI to do so before 2025.

@VictorLi not even humans can pass it. 700 is a mediocre score

@MarkIngraham plenty of people get 800 on the english score

@VictorLi a few thousand people have ever done that and only once by chance

@MarkIngraham not true. a few thousand high schoolers every year do it, and adult sat tutors do it consistently. (they brag about it to attract clients)

@MarkIngraham um no? i already have a couple of friends who got it last year and personally i only missed it by one question. its hard but definitely not impossible? where did u get ur stats?

@TommyMorriss sat tutors are considered scams by college board

@victor 1% get 1550 and if of 1.7 million .1% get 1600 that's 119,000 over the life of the test, and getting a perfect score assuming you get 40 of 50 questions and guess the rest is 0.1% by chance.

@VictorLi also its impossible to "miss by one question", you're allowed to miss several questions so you actually missed about five but the norm for that year was favorable.

@MarkIngraham u do realize how big a number 119,000 is right? im only asking for an AI that can do it once. out of the roughly 2 million students who do it per year roughly 500 get a perfect score so its far from impossible.

@MarkIngraham I know two people who got perfect scores, I only missed 7 questions. My tutor was significantly better than me, there is no doubt in my mind that he could ace it. We did practice sessions in parallel, I would miss two, he would miss zero and be finished in half the time. Granted, he knows all the tricks, but thats the point: you can master this skill like any other.

@MarkIngraham no. you can. i saw my score report from the College Board itself.

@VictorLi @MarkIngraham you get both numbers, your score/1600 (curved), and your question score. My year there was 0 points of wiggle room for an 800 on math, my friend's year there was 2 points. I think you have to pay extra to get question-level information, but I certainly got it.

@TommyMorriss 800 math is easy, it's the verbal that's subjective and useless

@TommyMorriss so you got a perfect score? Congrats but it happens to .1% of people by chance.

@VictorLi 119,000 out of the 119 million that took it since 1956, and it's actually 35,000. And if you eliminate half the questions and get the average white score you get that by chance.

@MarkIngraham why would i want the average white score?

@VictorLi because it's 100% luck, if you have 2000 friends 2 of them get perfect scores.

@MarkIngraham also i dont really care if its "subjective and useless", the point is whether or not an AI can do it before 2025. please dont entangle ur personal emotions with a purely technical timelines market

It would not be able to better than chance, if it did it would be a hyperintelligent AI that would kill all humans.

@MarkIngraham i can assure u it is not, coming from a high school senior himself. most of the perfect scorers i know have consistent gotten perfect scores on the practice tests, and like @TommyMorriss said many personal tutors can do it consistently themselves. getting a perfect on the SAT is not very difficult the main question is whether or not they can finetune it enough to do it

@MarkIngraham no, it will not be smart enough to kill all humans, it takes a lot more than a perfect SAT score to destroy humanity, believe me.

@VictorLi https://www.albert.io/blog/sat-score-calculator/ you can try here. Raw score 36 is a 600. Two questions free so 38. Then you guess 10 by chance. Then correct for two questions not reached. Then you need to guess two more.

@VictorLi your anecdote proves by point and it's only possible by chance.

@MarkIngraham i literally used this same one when prepping for the SAT myself, i know how it works. i really dont want to drag this conversation out any longer right now so if u believe it can't be done then feel free to bet NO and make some profit

@VictorLi OK well I already bet no, I'm just skeptical of your claims as I statistically proved it happened by chance and you can't do statistics yet have a perfect sat score.

@MarkIngraham u havent bet NO yet

@VictorLi if ur confident in ur claims then put ur mana where ur mouth is and bet NO, since the market is over 80% right now u can make some pretty sweet profits so why not?

@VictorLi d'oh I hit the wrong button and ran out of mana

I did not ace it, 7 questions off was far from a 1600 my year. look at some recent curves here:
https://old.reddit.com/r/Sat/comments/u48fol/official_march_23_2022_sat_score_release_and/

-7 was above average for me. on practice tests i would usually get -12 or -10, i did like 4 full ones

@TommyMorriss just retake the test 4000 times and get a perfect score by chance then only publish that result

@MarkIngraham I mean there is some chance, but its silly to think it doesn't correlate. If I took it 4000 times I would probably get a 1600. I took it 4 times and got within 40 points of myself each time

@MarkIngraham 🤣