Despite the hype, and compared to its competitors, Claude AI underperformed in managing the very documents it was designed to process.
Executive summary
- Claude AI struggled to process both a screenplay and a simple dialogue transcript.
- Even basic text input led to disorganized responses, often cluttered with emojis.
- The pro version’s pricing was vague, with no clear value proposition.
- Frequent interruptions and refill delays further undermined the experience.
- Ultimately, Claude AI failed to meet my expectations for handling even straightforward documents.
Introduction
A lot is happening in the AI space, with discussions, articles, commentary, as well as tips, tricks, and advice on the latest advancements.
This is how Claude AI came onto my radar.
Claude is promoted as some kind of intelligent document assistant that can read big files for you and extract key insights. It is equipped with advanced natural language processing (NLP) capabilities that allow it to read and interpret large documents effectively. This feature is supposed to enable users to extract key insights and summaries from extensive texts, which can be particularly beneficial in professional settings where time and accuracy are critical.
Some claim that Claude is better, faster, and more precise than, for example, ChatGPT and Gemini.
… Sigh… I don’t know.
Testing Claude AI with Real Documents
To put Claude to the test, I uploaded the screenplay of ‘Three Days of the Condor’ (1975) by Lorenzo Semple Jr. and David Rayfiel.
Claude couldn’t even read it.
The screenplay follows a standard 12-point script format, making it easy to read, yet Claude struggled. At 131 pages, it is a somewhat lengthy document, but handling such material shouldn’t be beyond an AI’s capabilities, even with complex dialogue and narrative structures.
The screenplay uses a standard 12-point font, which makes it easy to read and follows typical script formatting. The document has 131 pages, which supposedly makes it quite large for an AI to process at once, especially when extracting insights from complex dialogue and narrative structures.
The upload process stalled at “77%” and I got messages from Claude indicating that the document was too large.
Screenplays are simpler than many other types of content
This surprised me. I choose a screenplay for the reason that when compared to manuals, technical documents, legal texts, or the Bible, it becomes evident that these types of content are often thicker, lengthier, and more complicated.
Screenplays are primarily designed for storytelling and visual interpretation. They focus on dialogue and action. The language is typically concise and aimed at guiding performances. Screenplays prioritize clarity and brevity.
Attempting Simpler Documents and Conversations
I replaced the feature film screenplay and tried a dialogue transcript from a Hallmark Channel movie. These transcripts are significantly shorter and less complex, with simple dialogue and minimal descriptive content, making them much easier to process.
Claude AI struggled.
These are documents that ChatGPT and Gemini can easily chew up and spit out without any issues. Gemini even does it in the free version.
I compromised and tried a simple conversation without any documents or files. I copied and pasted a brief text from a press release and attempted to engage with Claude. The answers were disorganized. If the original text used emojis, Claude used them too, adding unnecessary noise which made it harder to read and comprehend.
The Problem with Eye Fixations and Disorganized Text
I am a quick reader, relying on eye fixations to briefly focus on specific words or phrases in a text. Scattered symbols and disorganized text can disrupt these fixations, making it difficult to efficiently process information at high speed. Claude AI’s scattered way of communicating in this test conversation forced me to repeatedly slow down and resort to a “sorting” process instead of focusing on comprehension.
Issues with the Pro Version and Pricing
Instead of delivering results, Claude kept nudging me to upgrade to the pro version. I was tempted by the vision and the enticing claims.
However, I felt that I should have been given a proper demonstration in the free test version. Crippleware disguised as you-get-what-you-pay-for underperformance shouldn’t be the reason to upgrade to a ‘better’ version. I had already wasted time — now I feared I might be wasting money too.
I gave it one last attempt in my exploration. I examined the pricing plan.
Instead of convincing me, the pricing structure was vague, and I had no clear idea of what I would be getting for the money if I subscribed.
The pro version might be great, but I could also be buying a pig in a poke. (The idiom “buying a pig in a poke” means purchasing something without seeing or fully understanding it first).
That ultimately made the decision for me. I opted to not proceed further.
Running Out of Patience and “credits”
Soon enough, I ran out of …something.. and was notified I would have to wait 3—4 hours for some sort of refill before I could try again.
The whole thing started to feel like a big joke.
If Claude AI had at least been able to manage relatively simple documents, then I would have easily subscribed. But what professional person or firm would need a system that seems barely capable of processing grade school papers?
In my experience, Claude AI failed to upload a screenplay for an average-length feature film, as well as a simple transcript of the words spoken between actors in a TV movie.
Imagine trying seriously big Document Types
In contrast to screenplays, the people who may need an advertised service like Claume are people who need to sift through for example manuals and technical documents; material created to convey detailed information about processes or systems. Manuals often include intricate explanations, specifications, and step-by-step instructions. The complexity arises from the need to cover all possible scenarios and ensure that users can effectively utilize the information. This results in a denser format that often requires specialized knowledge to fully comprehend. This is far from as straightforward as everyday conversations.
Legal texts also present another layer of complexity. They are crafted with precision to address specific legal standards, rights, and obligations. The language used is often formal and laden with jargon that can be challenging for laypersons to navigate. Legal documents must account for various interpretations and implications, which contributes to their length and intricacy.
The Bible stands out as a historical and theological document with layers of meaning that have been interpreted in numerous ways over centuries. Its narratives, poetry, and laws encompass a wide range of themes and contexts, making it both rich in content and complex in interpretation. The depth of its language can challenge readers, requiring careful study to grasp its full significance.
If you can’t process those types of documents, then what’s left? Maybe novels would be practical to be able to process with an AI assistant, sort of like having one’s own little private book club — someone to discuss the story with. Sadly, I got the impression Claude AI can’t cut it for such purposes.
Final Thoughts
Despite the evolution of Claude AI through its various versions, from Claude 1 to the latest Claude 3.5 Sonnet, the core issues remain unaddressed. The inability to handle relatively simple documents, unreliable performance, and vague pricing leave much to be desired. While improvements may have been made, my experience with Claude suggests that it has yet to reach a level of reliability and capability that would justify further investment.