What possible law would they be breaking?

LLMs have already been tricked by various PEs to give sensitive info or instructions for illegal things. Thus far they've been largely white hats. Use your imagination :)

@A7om20 Is the question whether a crime will be committed with the help of an LLM and then involved in the legal system, or if the prompting itself will be illegal(ie because of new laws)? Does it matter if someone is only charged but the court proceedings haven't yet begun?

@Aboczjr I think @A7om20 might be thinking something more like asking ChatGPT to help you commit a crime, where even planning to commit the crime is illegal, so it'll be the conversation with ChatGPT itself that would be illegal. Maybe if legal systems start to treat LLMs as if they're people it might count as a conspiracy, that sort of thing.

Or like, some content is illegal in various places - perhaps asking LLMs to generate such content will be illegal as well.

@chrisjbillington yea these are the type of things that would resolve to a yes. I doubt only prompting for instructions that could be used to commit crimes would prosecutable unless intent is pretty apparent @Aboczjr

Is market on a prosecution being filed, being filed and going to completion (regardless of outcome), or only if the prosecution is successful?

@DaveK prosecution does not need to be successful. Charges just have to be filed for a "yes" resolution.