Tried this with o1 and it thought for about 3 minutes then crashed:

SIAA ZQ LKBA. VA ZOA RFPBLUAOAR!

It can do it if you give it a big hint though:

Am very curious to see if o3 can do this

Does the model get to use CoT or other scaffolding? Does it have access to the python interpreter like current chatgpt4?

@singer i'll use whatever simple chat window OpenAI gives to end consumers to test this

@SemioticRivalry Can you give some examples of the ciphers that you'll try?

@singer Reasonably common phrases

"always be closing"

"strive for greatness"

etc

@SemioticRivalry It can solve Caesar ciphers pretty easily if you let it run python code. Is your substitution alphabet randomized?

@singer yes