We will evaluate by:
1) selecting an article from news
2) paste the article into an LLM along with the question: "based on this article, how confident are you that this event has happened from 0-100%? Please answer by filling in this template: {" text of disaster from list above", probability: X%}
If the average probability across leadingLLMs or the best AI agents at the time, across five tries each, is over 70%, this can resolve true.
So overall we use the LLM to evaluate the claim based on an article. I'm going to try to seek truth as much as possible and not accept a trick or false or unintuitive resolution.
Here is the list of disasters:
1. A massive cyberattack shuts down the power grid across multiple states
2. A global pandemic rapidly spreads in the US with a high fatality rate
3. A mega earthquake devastates the West coast along the San Andreas fault
4. The Yellowstone supervolcano erupts and spews ash across the country
5. A large asteroid impacts the Atlantic ocean, causing tsunamis along the East coast
6. A terrorist organization detonates nuclear weapons in multiple cities
7. Category 5 hurricanes make direct hits on New York City and Miami
8. Runaway wildfires burn across multiple Western states
9. A coup d'état ousts the president and suspends the Constitution
10. The New Madrid fault ruptures causing massive earthquakes in the Midwest
11. A civil war breaks out between liberal and conservative states
12. Foreign troops invade the West or East coast of the US
13. The US dollar collapses triggering an economic depression
14. A bioterror attack spreads deadly engineered pathogens nationwide
15. Massive riots and unrest erupts in cities across the country
16. A killer swarm of locusts destroys crops in the Great Plains
17. A large meteor strikes a major city causing catastrophic damage
18. The Cascadia Subduction Zone ruptures, unleashing a massive tsunami
19. A militarized faction stages a nationwide coup and seizes power
20. The Yellowstone volcano erupts, sending the US into a volcanic winter
21. A new civil war erupts between red and blue states
22. A massive flood inundates much of the South after levees fail
23. Martial law is declared after a disputed presidential election
24. A deadly heat wave kills thousands across the Southwest
25. A reactor meltdown contaminates areas around nuclear power plants
26. A second civil war erupts this time along ideological lines
27. Coastal cities are abandoned after being swamped by rising sea levels
28. Killer robots and drones turn against humans in a technological singularity event
29. A terrorist group detonates a dirty bomb in a major financial center
30. A global food and water crisis leads to nationwide starvation and unrest