ChatGPT and all its ilk *always* make something up.

That's literally all they do.

They just come out with "what's statistically likely to be the stream of words/images/whatever that would follow a question that looks like that". The miracle is that sometimes, the pig manages to sing a recognisable tune; the sad part is that so many people assume that because it's right sometimes, it's going to be right more often ...

This is the same quality of thinking that leads to "mediums" talking to your dead relatives.