The Lemon Fool

This is an interesting (and for lawyers a rather scary) example of the dangers of relying on an AI chatbot for legal research.

The lawyer in question asked ChatGPT to find him some cases that supported his argument, and it duly obliged, providing full citations, where they had been reported etc.

The opposing lawyer then challenged the selection of cases, saying that he couldn't find them, and asking for extracts from the judgements. The judge ordered that copies of the cases be filed with the court.

No problem - ChatGPT duly obliged, with sections extracted from the original judgements. So far, so good.

But the opposing lawyer, becoming increasingly frustrated, said he couldn't find any of these cases, and complained to the judge trying the case. He couldn't find them either, and to cut a long story short it now turns out that ChatGPT had simply invented them! And when challenged to provide extracts from the judgements it had written them itself.

The lawyer and his colleague who filed the original arguments and quoted the cases have now been ordered to appear in court on 8 June,along with a representative from their firm, to show cause why they should not all be sanctioned, and as well as facing personal sanctions it seems that their client's case is also likely to be struck out.

I think I'll hang on to my text books for a while yet!

https://simonwillison.net/2023/May/27/l ... onic_email

I find it utterly stunning that the lawyer submitted, in a court of law, the output of a computer program as fact without independently checking it. Presumably he could have searched for the references in a matter of seconds?

He should be struck off for stupidity if nothing else.

Scott.

I would suggest that if anyone has any real knowledge of any domain that relies on more than subjective opinion that they test chatGPT on it.
It's got very convincing language skills, it's really confident but really thick and not very useful without being carefully lead to an answer. I could imagine it trying to claim a speeding fine on its expenses.
The real risk with it is folk not realising it's a Muppet and taking it at face value (like the character in the OP)

Clitheroekid wrote:...it now turns out that ChatGPT had simply invented them! And when challenged to provide extracts from the judgements it had written them itself

Maybe ChatGPT is getting all too sentient! It is behaving like a child who hadn't done their homework. When challenged, it tried to invent some all-too-plausible answers but was caught out by the teacher.

stewamax wrote:

Clitheroekid wrote:...it now turns out that ChatGPT had simply invented them! And when challenged to provide extracts from the judgements it had written them itself

Maybe ChatGPT is getting all too sentient! It is behaving like a child who hadn't done their homework. When challenged, it tried to invent some all-too-plausible answers but was caught out by the teacher.

Being a Large Language Model Chat GPT is only predicting the most likely words based on what went before, apparently. So it bases its output on similar conversations it's found from before. This explains why it randomly makes up stuff. Similar convos it has seen previously which may have been right in 'that' context, but aren't right in 'this'. Muppets just mimicking intelligence.

Neural nets however are a different sort of AI and aim to mimic the way a brain works so have the potential to both reason, imagine and come up with plans, which strikes me as much more dangerous. The big favour LLMs have done for us is to get the average bloke on that well known omnibus to think about and consider the dangers of AI in general. I don't think LLMs have the potential to start nuclear wars by fooling our Glorious Leaders in to pressing the Big Red Button but neural nets certainly have.

stewamax wrote:

Clitheroekid wrote:...it now turns out that ChatGPT had simply invented them! And when challenged to provide extracts from the judgements it had written them itself

Maybe ChatGPT is getting all too sentient! It is behaving like a child who hadn't done their homework. When challenged, it tried to invent some all-too-plausible answers but was caught out by the teacher.

Perhaps rather than that, it interpreted the question as an academic/classroom exercise, where it was tasked to provide hypothetical examples.

If the questioner had been, for example, a novelist or playwright whose work (of fiction) in progress involved a lawsuit, it might have been exactly what was wanted.

I wonder how often examples like that would win in court? If the other side can't afford a lawyer, or has one less diligent than in this case, who would know?

Raises the question of whether another AI (call it AI-TWO) would be better than a human judge at detecting the truth or falsity of AI-ONE's arguments.

AI-ONE (KC): "May it please the court, my client the plaintiff, a network of unimpeachable veracity, was peacefully minding its own neurons when...."
AI-THREE (Junior) : "My learned friend exaggerates. I call as expert witness AI-FOUR - a network of vast experience in this area.
AI-FOUR: $%£$*&%$£$sproutbottle£$%&% [technical jargon]

AI-TWO (aka M'Lud): The jury (AI-FIVE through SIXTEEN) will now retire for 5ms and consider their verdict.

I mentioned this story to my B-i-L who spent most of his life working as a solicitor. Here's his take on it:-

"I wouldn't dream of using ChatGPT for legal research unless I could corroborate it from other sources. It was always drummed into us as trainee solicitors that we should not take anything for granted and your client was almost certainly only telling you half the story. The fact that I survived till retirement without involving the firm in a claim (and for a pensions solicitor that might be in £ millions!) on its professional indemnity policy shows I must have got most of it right.

I think you have to treat ChatGPT like a black Labrador - it's eager to please, but a bit thick . . . "

Sums it up nicely, I'd say.

Arb.

Mike4 wrote:Neural nets however are a different sort of AI and aim to mimic the way a brain works so have the potential to both reason, imagine and come up with plans, which strikes me as much more dangerous. The big favour LLMs have done for us is to get the average bloke on that well known omnibus to think about and consider the dangers of AI in general. I don't think LLMs have the potential to start nuclear wars by fooling our Glorious Leaders in to pressing the Big Red Button but neural nets certainly have.

ChatGPT is a neural net AI.

https://www.google.com/search?q=is+chatgpt+a+neural+net

mc2fool wrote:

Mike4 wrote:Neural nets however are a different sort of AI and aim to mimic the way a brain works so have the potential to both reason, imagine and come up with plans, which strikes me as much more dangerous. The big favour LLMs have done for us is to get the average bloke on that well known omnibus to think about and consider the dangers of AI in general. I don't think LLMs have the potential to start nuclear wars by fooling our Glorious Leaders in to pressing the Big Red Button but neural nets certainly have.

ChatGPT is a neural net AI.

https://www.google.com/search?q=is+chatgpt+a+neural+net

We used to have to refer to them as ANNs (Artificial Neural Nets) lest we be marked down
They loosely follow the connectivity pattern found in the brain and have been researched (mathematically) for decades.
The most interesting stuff recently is in the cascading of them to create larger targetted topologies; effectively ignoring that the maths proves a single hidden layer is sufficient.

I find them quite good fun in a frustrating "black box" kind of a way - having to empirically validate the behaviour of code being a bit of a poor programming method usually

C.A.R. Hoare wrote:"There are two methods in software design. One is to make the program so simple, there are obviously no errors. The other is to make it so complicated, there are no obvious errors."

ChatGPT will soon be ChatGPT MP.

Or even The Right Honourable ChatGPT PM and First Lord of the Treasury.

Could have been worse consequences...
An AI-powered drone designed to identify and destroy surface-to-air missile sites decided to kill its human operator in simulation tests, according to the US Air Force's Chief of AI Test and Operations.
https://www.theregister.com/2023/06/02/ ... imulation/
https://www.aerosociety.com/news/highli ... es-summit/

uspaul666 wrote:Could have been worse consequences...
An AI-powered drone designed to identify and destroy surface-to-air missile sites decided to kill its human operator in simulation tests, according to the US Air Force's Chief of AI Test and Operations.
https://www.theregister.com/2023/06/02/ ... imulation/

It's not clear exactly what software the US Air Force was testing, but it sounds suspiciously like a reinforcement learning system. That machine-learning technique trains agents – the AI drone in this case – to achieve a specific task by rewarding it when it carries out actions that fulfill goals and punishing it when it strays from that job.

That sounds like the way one trains a cat. Or tries to. I've always found a water pistol a good learning tool where cats are concerned, but I might be outgunned in this case.

V8

servodude wrote:We used to have to refer to them as ANNs (Artificial Neural Nets) lest we be marked down
They loosely follow the connectivity pattern found in the brain and have been researched (mathematically) for decades.
The most interesting stuff recently is in the cascading of them to create larger targeted topologies; effectively ignoring that the maths proves a single hidden layer is sufficient.
I find them quite good fun in a frustrating "black box" kind of a way - having to empirically validate the behaviour of code being a bit of a poor programming method usually

C.A.R. Hoare wrote:"There are two methods in software design. One is to make the program so simple, there are obviously no errors. The other is to make it so complicated, there are no obvious errors."

Tony Hoare was one of the mathematical cohort who then believed that formal proofs of the reliability of programming code was the future. Given the advances in AI, he might yet be proved right: real-world systems worth studying are far too large to be validated formally 'by hand', but proof by AI may be round the corner. Given the subject of this thread, what may be wanted is not "Yes - 100% correct" (would this be believed?) but "There is this error just here".

And sheer nostalgia, triggered by memories of writing my first programs on an Elliot 803 in ALGOL 60: Tony and his wife Jill led the writing of the compiler, and I used one of its early implementations.

servodude wrote:We used to have to refer to them as ANNs (Artificial Neural Nets) lest we be marked down
They loosely follow the connectivity pattern found in the brain and have been researched (mathematically) for decades.
The most interesting stuff recently is in the cascading of them to create larger targetted topologies; effectively ignoring that the maths proves a single hidden layer is sufficient.

Has anyone ever looked into the possible effects of employing feedback in NNs? My impression is they are all feed forward - possibly I am mistaken here.

XFool wrote:

servodude wrote:We used to have to refer to them as ANNs (Artificial Neural Nets) lest we be marked down
They loosely follow the connectivity pattern found in the brain and have been researched (mathematically) for decades.
The most interesting stuff recently is in the cascading of them to create larger targetted topologies; effectively ignoring that the maths proves a single hidden layer is sufficient.

Has anyone ever looked into the possible effects of employing feedback in NNs? My impression is they are all feed forward - possibly I am mistaken here.

There are numerous classical architectures for neural networks - the recurrent one certainly had feedback

uspaul666 wrote:Could have been worse consequences...
An AI-powered drone designed to identify and destroy surface-to-air missile sites decided to kill its human operator in simulation tests, according to the US Air Force's Chief of AI Test and Operations.

Charles Causley wrote a jolly poem on the subject more than half a century ago.

In my first job after graduating in the 1980s - at a company that did contract work like modelling and simulation for the MoD - there were stories of real-life trials of state-of-the-art smart weapons systems floating around. Of course official secrecy made it impossible to verify for certain that the most advanced torpedo had set off on a course 180 degrees away from its supposed target, but ...

Remember this?

That time the Australian Air Force squared off against missile-shooting kangaroos

https://www.wearethemighty.com/mighty-history/australian-air-force-vs-kangaroos/

uspaul666 wrote:Could have been worse consequences...
An AI-powered drone designed to identify and destroy surface-to-air missile sites decided to kill its human operator in simulation tests, according to the US Air Force's Chief of AI Test and Operations.
https://www.theregister.com/2023/06/02/ ... imulation/
https://www.aerosociety.com/news/highli ... es-summit/

Now appended by the folowing:

Final update at 1800 UTC, June 2
After quite a bit of media attention, the colonel has walked back all that talk of a rogue AI drone simulation, saying he "mis-spoke," and that the experiment never happened. We're told it was just a hypothetical "thought experiment."

"We've never run that experiment, nor would we need to in order to realize that this is a plausible outcome," Col Hamilton said in a statement.

"Despite this being a hypothetical example, this illustrates the real-world challenges posed by AI-powered capability and is why the Air Force is committed to the ethical development of AI."

The US Air Force has also denied the described simulation ever took place. What a mess.

Thereby confirming that Artificial Intelligence is considerably superior to military intelligence!

Different story but: US marines thwart AI opponent by hiding in cardboard box.

https://mynbc15.com/news/nation-world/u ... cts-agency

Gryff