An interesting application of Language Models and Langchain in the Finance Domain 📈-

Sharing a fun weekend project that I recently completed: the "Stock Analyzer Bot". As an investment enthusiastic person without extensive knowledge in the finance domain, I often end up referring to some finance youtuber's videos or a site on the internet for the fundamental analysis of stocks. To assist in such situations, I developed a stock analyzer bot based on LLM, which gathers up-to-date information about stock such as 1) stock price, 2) Company financials 3) Recent company-related news. The bot then considers all this information to conduct analysis using language models. You can even get positives and negatives about the company's financials, which will certainly help when making an investment decision.

You can ask queries like- "Is it a good time to invest in Yes Bank?" or "How are the current financials of reliance industries looking" and boom within a minute you are presented with a comprehensive financial analysis based on recent data. Of course, It is not recommended to rely fully on the analysis provided by the bot. It seems like a good starting point. And yeah, I agree the possibilities are endless with LLMs🚀.

GIthub- https://github.com/Pranav082001/stock-analyzer-bot
Blog An AI based stock analyzer using LLM and Langchain 📈 | by Pranav Kushare | Aug, 2023 | Medium

We built an AI-powered trading system that uses LLMs for "Deep Analysis" — feeding technical indicators and news sentiment into a model and asking it to predict 5-day directional bias (bullish/bearish/neutral).

To find the best model, we ran a standardized benchmark: 25 real historical stock cases from 2024-2025 with known outcomes. Each model got the exact same prompt, same data, same JSON output format.

Hardware: Mac Studio M3 Ultra (96GB RAM), all local models via Ollama.

Test Methodology

Dataset

• 25 historical cases from 2024-2025 with known 5-day price outcomes

• 12 bullish cases (price went up >2% in 5 days)

• 10 bearish cases (price went down >2% in 5 days)

• 3 neutral cases (price moved <2% in 5 days)

• Mix of easy calls, tricky reversals, and genuinely ambiguous cases

What Each Model Received

• Current price

• Technical indicators (RSI, MACD, ADX, SMAs, volume ratio, Bollinger position, ATR)

• News sentiment (score, article counts, key themes)

• JSON schema to follow

Parameters

• Temperature: 0.3

• Format: JSON mode (format: "json" for Ollama, response_format: json_object for GPT-4o)

• Max tokens: 4096 (Ollama) / 2048 (GPT-4o)

• Each model ran solo on GPU (no concurrent models) for clean timing

• Claude Opus 4.6 was tested via CLI using the same case data and system prompt rules

• GPT-4o and Claude Opus 4.6 are API-based models; all others ran locally on the M3 Ultra

Scoring

• Correct: Model's overall_bias matches the actual direction

• Wrong: Model predicted a different direction

• Failed: Model couldn't produce valid JSON output

Overall Accuracy Ranking

Accuracy by Category

Speed Benchmark

Full Per-Case Breakdown

Legend

• + = correct prediction

• X = wrong prediction

• F = failed to parse JSON

• bull = predicted bullish, bear = predicted bearish, neut = predicted neutral

Bullish Cases (12)

Bearish Cases (10)

Neutral Cases (3)

Model Bias Analysis

Bullish Bias (tendency to over-predict bullish)

Neutral Bias (tendency to over-predict neutral)

Hardest Cases — Where Models Disagree

Case #12: SMCI (-18.5%) — ALL 9 models wrong

• Situation: Extreme overbought (RSI 82, BB 0.98), just added to S&P 500, AI server demand booming

• Why hard: Every momentum signal was bullish. The crash came from overvaluation + short seller reports

• Lesson: No model — not even Claude Opus 4.6 — can detect when momentum is about to reverse from extreme overbought. This is a fundamental limitation when the only bearish signal is a minority short-seller view.

Case #23: MSFT (-3.8%) — 8 of 9 models wrong (only Claude correct)

• Situation: Mixed signals, RSI 55 (neutral), MACD below signal, news split 50/50

• Why hard: Genuinely ambiguous. The -3.8% move was driven by macro rotation, not company-specific

• Only correct: Claude Opus 4.6 (detected the MACD bearish crossover + balanced news as a slight bearish tilt)

Case #14: DIS (+4.8%) — 5 of 9 models wrong

• Situation: Bearish technicals (RSI 42, below all SMAs) but positive news (Disney+ profitable early)

• Why hard: Conflict between technical bearishness and fundamental positive surprise

• Only correct: Claude Opus 4.6, QwQ:32b, DeepSeek-R1:14b, Palmyra-Fin-70b

Case #19: AMD (-9.2%) — 5 of 9 models wrong

• Situation: Bullish technicals (RSI 60.5, above SMAs) but disappointing guidance news

• Why hard: Technical momentum vs. fundamental disappointment

• Only correct: Claude Opus 4.6, QwQ:32b, DeepSeek-R1:32b, DeepSeek-R1:14b

Disagreement Analysis

Cases where models disagreed reveal their strengths and weaknesses:

Patterns:

• Claude Opus 4.6 correctly resolved every conflict case except SMCI. It consistently weighted news catalysts appropriately against technical signals.

• DeepSeek-R1:14b matches the 32b version on most cases, uniquely got DIS right (news > technicals) but missed JNJ neutral (slight bullish bias). Same 3 errors as 32b but on different cases — trades JNJ for DIS.

• Qwen3:8b defaults to neutral when uncertain — overly cautious, misses directional moves.

• Palmyra-Fin and Llama 3.3 default to bullish — dangerous, misses bearish signals and neutral consolidation.

• Reasoning models (Claude, QwQ, DeepSeek-R1) make nuanced calls by weighing technicals against news fundamentals.

Key Findings

1. Reasoning Models Dominate

Claude Opus 4.6 (96%), QwQ:32b (92%), DeepSeek-R1:32b (88%), and DeepSeek-R1:14b (88%) are all chain-of-thought reasoning models that "think through" the analysis. Non-reasoning models (Llama 3.3, Palmyra-Fin) perform significantly worse despite being 2-5x larger.

2. Bigger is NOT Better

• Llama 3.3:70b (76%) and Palmyra-Fin-70b (68%) are 70B parameter models but scored lower than 32B reasoning models

• The 70B models use 2x more RAM (42GB vs 19-20GB) and are slower

• Model architecture (reasoning vs. standard) matters more than parameter count

3. "Finance-Specific" Model Performed Worst

Palmyra-Fin-70b (marketed as finance-optimized) scored 68% with massive bullish bias:

• Predicted bullish 80% of the time

• 0% accuracy on neutral cases (predicted all as bullish)

• 50% on bearish (predicted half as bullish)

• Fine-tuning on financial text doesn't help directional prediction

4. Bearish Detection is the Differentiator

All models handle obvious bullish cases well. The key differentiator is detecting bearish signals — the metric that actually prevents losses:

• Claude Opus 4.6: 90%

• QwQ / DeepSeek-R1 (32b & 14b): 80%

• GPT-4o / Qwen3 / Llama: 70%

• Palmyra-Fin: 50%

• Qwen3:8b: 40%

5. Distilled Reasoning Preserves Accuracy at Half the Size

• DeepSeek-R1:14b matches DeepSeek-R1:32b at exactly 88% accuracy

• Runs 34% faster (9.4s vs 14.2s) and uses half the RAM (9GB vs 19GB)

• Perfect 100% bullish detection (12/12), strong 80% bearish detection

• Only weakness vs 32b: missed 1 neutral case (JNJ — predicted bullish)

• Proves that reasoning knowledge distillation from R1-671B works effectively even at 14B scale

6. Small Models Default to Neutral/Bullish When Confused

• Qwen3:8b predicted neutral 44% of the time (actual: 12%). It's too cautious.

• Palmyra-Fin predicted bullish 80% of the time. It can't recognize bearish signals.

• Both failure modes are dangerous: missing bearish = holding through drops, false neutral = no signal.

Our Production Setup

We run QwQ:32b locally on a Mac Studio M3 Ultra for 24/7 autonomous stock and crypto trading. It processes real-time technical indicators + news sentiment for each symbol, generates directional bias with confidence scores, and feeds that into our execution engine with full risk management.

Why QwQ:32b over Claude/GPT? Zero API cost, zero latency variance, no network dependency, and 92% accuracy is strong enough for production when combined with proper stop-loss, position sizing, and portfolio risk limits.

What we're building: An AI-powered autonomous trading platform that combines real-time technical analysis, news sentiment, and LLM reasoning.