AI

LLMs are totally unreliable for research. They are just probable token generators.

Especially if your looking for new data that nobody has talked about before, then your just going to get convincing hallucinations, like talking to a slightly drunk professor at a loud bar who can't ever admit they don't know something.

Example: ask a llm this "what open source software developer died in the September 11th attacks?"

It will give you names, and when you try to verify those names, you'll find out those people didn't die. It's just generating probable tokens

Absolutely not.

So are LLMs reliable for research like that?

No. Of course not. They're not reliable for anything. They don't have any kind of database of facts and don't know or attempt to know anything at all.

They're just a more advanced version of your phone's predictive text. All they do is try to figure out which words most likely go in what order as a response to the prompt. That's it. There is no logic of any kind dictating what an LLM outputs.

Treat it like an eager impressionable intern with a confident stride.

They aren’t. They’re a party trick.

If you're using an LLM, you should limit the output via a grammar to something like json, jsonl, or csv so you can load it into scripts and validate that the generated data matches the source data. Though at that point you might as well just parse the raw data and do it yourself. If I were you, I'd honestly use something like pandas/polars or even excel to get it done reliably without people bashing you for using the forbidden technology even if you can 100% confirm that the data is real and not hallucinated.

I also wouldn't use any cloud LLM solution like OpenAI, Gemini, Grok, etc. Since those can change and are really hard to validate and give you little to no control of the model. I'd recommend using a local solution like running an open weight model like Mistral Nemo 2407 Instruct locally using llama.cpp or vLLM since the entire setup will not change unless you manually go in and change something. We use a custom finetuned version of Mixtral 8x7B Instruct at work in a research setting and it works very well for our purposes (translation and summarization) despite what critics think.

Tl;dr Use pandas/polars if you want 100% reliable (Human error not accounted). LLMs require lots of work to get reliable output from

Edit: There's lots of misunderstanding for LLMs. You're not supposed to use the bare LLM for any tasks except extremely basic ones that could be done by hand better. You need to build a system around them for your specific purpose. Using a raw LLM without a Retrieval Augmented Generation (RAG) system and complaining about hallucinations is like using the bare ass Linux kernel and complaining that you can't use it as a desktop OS. Of course an LLM will hallucinate like crazy if you give it no data. If someone told you that you have to write a 3 page paper on the eating habits of 14th century monarchs in Europe and locked you in a room with absolutely nothing except 3 pages of paper and a pencil, you'd probably write something not completely accurate. However, if you got access to the internet and a few databases, you could write something really good and accurate. LLMs are exceptionally good at summarization and translation. You have to give them data to work with first.

I work on a project where we are trying to analyse financial data. We use Claude and llama and they are really good. We needed a few months to achieve 87% reliability.

For our application that's probably almost enough. For an application that needs 100% all the time, every time, that quite a lot away.

How reliable is autocorrect?

They are about as reliable as any random person online. If you wouldn't trust the data from a person posting on social media, don't trust the LLM.

If generation temperature is non-zero (which it often is), there is inherent randomness to the output. So even if the first number in a statistic should be 1, sometimese it will just randomly pick any other plausible number. Even if the network always picks the correct token as the highest probability, it's basically doing a coin toss for every token to make answers more creative.

That's on top of hoping the LLM has even seen that data during training AND managed to memorize it during training AND that the networks just happens to be able to reproduce the correct data given your prompt (it might not be able to for a different prompt).

If you want any reliability at all, you need to use RAG AND also you yourself have to double check all the references it quotes (if it even has that capability).

Even if it has all the necessary information to answer correctly in it's context window, it can still answer incorrectly.

None of the current models are anywhere close to producing trustworthy output 100% of the time.

It's a scam.

The least unreliable LLM I've found by far is perplexity, in the Pro mode. (By the way, if you want to try it out, you get a few free uses a day).

The reason is because the Pro mode doesn't retrieve and spit out information from its internal memory bank, but instead, it uses that information to launch multiple search queries, then summarises the pages it finds, and then gives you that information.

Other LLMs try to answer "from memory" and then add some links at the bottom for fact checking but usually Perplexity's answers come straight from the web so they're usually quite good.

However, I still check (depending on how critical the task is) that the tidbit of information has one or two links next to it, that the links talk about the right thing, and I verify the data myself if it's actually critical that it gets it right. I use it as a beefier search engine, and it works great because it limits the possible hallucinations to the summarisation of pages. But it doesn't eliminate the possibility completely so you still need to do some checking.