01 / The question
Does financial text add information the numbers miss?
We investigated whether sentiment from financial text improved stock-direction prediction beyond financial statements and market behavior. The useful question was whether another data source actually added predictive value, once we tested it on future observations.
We worked with 34 Consumer Staples tickers, using SEC fundamentals, adjusted price and volume data, SEC filing text, and Capital IQ Key Developments. Keeping the universe within one sector made the comparison more controlled, but limited what we could say about the rest of the market. Report, III.A ↗
The result in context
The strongest reported sentiment configuration reached 0.6038 holdout AUROC. Its mean cross-validation AUROC was 0.4989.
That gap is the point of this case study. The result was worth investigating, but one better year did not establish a dependable signal.
02 / Building the experiment
Three layers. Three different reasons a prediction might improve.
The pipeline added feature groups in stages. That gave us a way to compare accounting information, market context, and text sentiment instead of treating the final model as one unexplained score.
01Financial fundamentalsWhat does the company's financial position look like?
SEC 10-Q and 10-K data became liquidity, debt, profitability, growth, and cash-flow ratios. Unit inconsistencies, near-zero denominators, and mismatched accounting periods could make those features misleading before a model ever saw them.
The report records a fundamentals rebuild that reduced Universe V1 feature-ratio sanity flags from 2,966 to 63. Those are diagnostic flags, not a claim that every remaining value was correct. Report, III.C ↗
02Market behaviorWhat was happening before the prediction?
Price and volume data supplied momentum, realized volatility, drawdown, abnormal volume, and relative-strength features. The timing rule mattered: a backward-looking feature must stop before the prediction date.
In the reported course ladder, adding market features increased mean CV AUROC from 0.4878 to 0.5304 but reduced holdout AUROC from 0.5501 to 0.5020. It was an addition to the feature set, not an automatic improvement. Report, IV.A.2 ↗
03Financial text sentimentWas the event's tone different from its peers?
Pretrained FinBERT converted financial text into positive, negative, and neutral sentiment measurements. Those measurements and text counts became inputs to the downstream classifiers. Sector-relative sentiment subtracted the sector's contemporaneous average tone from a company's tone.
Capital IQ Key Developments supplied event text with better coverage than the sparse external-news windows we evaluated. The report lists 92.70% coverage for its 63-day 2024 holdout window. That is text availability, not predictive accuracy. This path used sentiment feature engineering; sector-specific FinBERT fine-tuning was a separate research idea. Report, III.C; project history ↗
Why the dates mattered more than the model choice
A fiscal quarter ending does not mean its financial report was public that day. The event pipeline needed a public-availability date, a prediction date, features known before that prediction, and labels measured afterward.
- Fiscal period endsThe quarter is over; the filing may not be available yet.
- Filing becomes publicIts information can enter the availability calculation.
- Prediction dateUse eligible prior features; respect filing time and the trading calendar.
- Future outcome windowMeasure the return being predicted, separately from the inputs.
Validation used expanding chronological folds, purging of overlapping outcome windows, an embargo, and a separate 2024 holdout. Preprocessing also had to learn its imputation and scaling rules from training rows. These were design requirements and audited checks, not proof that every possible leakage risk had been eliminated. Report, II–III; rebuild code ↗
03 / Working with ChatGPT and Codex
The questions and corrections that moved the project.
I used ChatGPT to understand the methods, compare options, and prepare bounded prompts. I took those prompts into Codex for implementation and brought results back for review. That helped me build beyond what I could have written independently, but it also meant I needed to check the advice against the actual work.
These are excerpts from my project messages. The explanations below them are retrospective paraphrases, not reconstructed dialogue.
March 182026Start with a bounded build
“so lets begin with layer 1 and ill proceed with 2015-2024, as long as I can add data to it”
My message · Project Start GuidanceI chose a manageable starting window and asked for prompts I could take into Codex. ChatGPT suggested one bounded milestone at a time, beginning with SEC fundamentals before the other layers.
What followed
The implementation developed in steps: data collection, ratio construction, panel assembly, and a baseline. This was an AI-assisted workflow, with me directing the build and asking questions as I went.
Evidence: Project Start Guidance · user questions and ChatGPT replies.
April 52026Give the research assistant the actual scores
“i like it, but why wouldnt we also give it the scores?”
My message · Supervised Learning PipelineChatGPT drafted a research prompt around the project state. I pointed out that the model scores belonged in that prompt too, then supplied the comparison table.
How the advice changed
ChatGPT revised its advice to use AUROC, F1, precision, recall, log loss, and feature counts as diagnostic evidence. The next research brief carried the results, rather than just saying a layer had lost.
Evidence: Supervised Learning Pipeline · user correction and ChatGPT response.
April 222026Correct what we were predicting
“I want to do quarterly predictions not 5 day predictions.”
My message · Quarterly Prediction RefactorI stopped to correct the target. We compared a fixed 63-trading-day window with a variable filing-to-next-filing window. ChatGPT recommended the fixed horizon as a cleaner definition, not evidence of higher realized performance.
What followed
The preserved course configuration names a 63-day sector-relative sign target. The supplied report still contains 21-day wording, so the documentation did not stay fully synchronized with the code.
Evidence: Quarterly Prediction Refactor · user correction; preserved 63-day config.
April 252026Check whether more text was actually usable
“so we are running another audit?”
My message · Project Data Layer AuditA large external news corpus looked useful at first. ChatGPT distinguished a broad access/schema check from a targeted test of ticker and event-window coverage. I brought the audit results back for review.
What followed
The targeted audit still recommended against integration. I described to my teammate why we would not use most of that corpus. Dataset size alone was not a reason to run more sentiment inference.
Evidence: Project Data Layer Audit · user audit readouts and ChatGPT review.
April 272026Ask whether adding a layer guarantees a gain
“does each layer build off each other to boost predictive power?”
My message · FinBERT Sector-Tuning IntegrationChatGPT clarified the difference between adding features and demonstrating incremental predictive value. The architecture was additive; the results did not have to improve in that order.
What followed
That distinction is visible in the course ladder: market features improved cross-validation but hurt the 2024 holdout, while sentiment improved the holdout without matching stability across validation periods.
Evidence: FinBERT Sector-Tuning Integration · user question and ChatGPT clarification.
April 302026Correct the presentation before it overstated the work
“This is still the proposed framework. We are not done”
My message · Poster Presentation ScriptI took a ChatGPT presentation script through Codex against the source material, then returned with corrections. I explicitly noted that parts of the poster were a proposed framework.
What followed
The revised script separated designed methods from completed experiments. The same distinction matters here: optional models, tuning ideas, and planned SHAP plots are not completed accomplishments.
Evidence: Poster Presentation Script · user correction after a Codex review.
May 28–292026Turn the follow-up into explicit gates
“This will be a new branch. Give me a comprehensive milestone based sequence I can act on. I will be using this chat as a QA for any help”
My message · Event-Level Stock PredictionI asked ChatGPT for the next branch plan and carried its milestones into Codex. The later rebuild used a 21-day primary horizon, with 10- and 63-day checks, and kept validation ahead of model comparisons.
What followed
Codex implemented the modules, tests, and reports. Its ablation used a fixed logistic learner to isolate feature changes. The saved results were weaker than the course result; they remain a separate experiment below.
Evidence: Event-Level Stock Prediction · ChatGPT plan; Define branch contract · Codex execution.
04 / What the results support
A better holdout score was only part of the answer.
Switch between metrics to see why the highest AUROC did not settle the question. Select a configuration for its interpretation, or inspect the full table. The later rebuild is included separately because it changed the experiment and produced a different result.
Reported course ladder / 2024 holdout
34 Consumer Staples tickers. The final preserved configuration specifies 63 trading days; the supplied report retains conflicting 21-day wording. Each rung reports its selected learner, so this is not a fixed-model ablation.
Configuration D / Logistic regression
+ Sector-relative sentiment
The highest reported holdout AUROC, but cross-validation remained near 0.50 and F1 was the lowest of the four. This is the result that needed qualification.
Inspect the numbers and their source
Transcribed from the supplied report, IV.A.2, Capital IQ Layered Sentiment Ladder. CV means cross-validation. AUROC measures ranking discrimination, not percentage accuracy. F1 depends on the classification threshold; log loss evaluates probability quality.
| Configuration | Mean CV | Worst fold | Holdout | Log loss | F1 |
|---|---|---|---|---|---|
| A · Financial fundamentals | 0.4878 | 0.3872 | 0.5501 | 0.7289 | 0.5421 |
| B · + Market behavior | 0.5304 | 0.4083 | 0.5020 | 0.7776 | 0.5357 |
| C · + Raw sentiment | 0.4927 | 0.4012 | 0.5958 | 0.6939 | 0.4792 |
| D · + Sector-relative sentiment | 0.4989 | 0.4203 | 0.6038 | 0.7115 | 0.3765 |
These controls explore saved results. They do not train a model or generate new predictions.
How I would read the course result
Sector-relative sentiment raised the reported 2024 holdout AUROC from 0.5020 to 0.6038 compared with core + market, a difference of 0.1018 AUROC. That is not a 10.18% accuracy improvement. The selected model also changed from XGBoost to logistic regression, which limits a feature-only interpretation.
The companion report synthesis records mixed pseudo-holdout years and a bootstrap AUC-delta interval of −0.0930 to 0.1307. The available synthesis does not establish that this is a confidence interval for the selected-rung difference above. It crosses zero, and the supplied original leaves its statistical-analysis section unfinished. I cannot claim a completed significance result. Report versions and stability evidence ↗
What the follow-up did not confirm
The later fixed-learner rebuild did not reproduce a strong 21-day full-stack result: holdout AUROC was 0.3412 against a naive 0.5000 reference. Its saved ranking diagnostic also put the full stack's highest-score decile at −0.0296 average excess return. These are archived diagnostics from a different setup, not an independent replication of the course experiment or an executable trading backtest. May 29 saved diagnostics ↗
05 / My role in the build
The architect and the manager.
Working with advanced AI changes where I can contribute. I can work at the level of the system: define the question, organize the pieces, and decide what needs to happen next. ChatGPT helped develop the approach, and Codex wrote and revised substantial code. My responsibility was to keep that work connected to the research question and the evidence.
Architect: define the system
The target, timing rules, and comparison determine what a model's score means. Correcting the prediction horizon changed the question the pipeline needed to answer. AI helped develop the design; I needed to understand enough of it to catch a mismatch and redirect the work.
Manager: make the work reviewable
I asked for bounded builds, brought results back into the research prompts, and checked whether the presentation matched what was finished. Breaking the project into stages gave me places to review the output, ask questions, and change direction before adding more complexity.
Reviewer: decide what the evidence supports
Producing a working pipeline and establishing a useful signal are different tasks. A higher holdout score still needed context: cross-validation, coverage, changing learners, and the prediction horizon. Directing the build also meant deciding which claims we could make from it.
What this means as AI becomes more capable
The philosophy I take forward is that delegating implementation still requires understanding. I need enough understanding to specify the goal, question an answer, and recognize when the system is solving the wrong problem. AI can help with that understanding too, but I still have to make and defend the decisions.
If AI progresses toward superintelligence (SI), I see the architect and manager role becoming more important: choosing worthwhile problems, setting constraints, and deciding what counts as success. That is a direction for how I want to work. This project was practice with today's tools, where those responsibilities were already concrete.
Limits I would keep in a future version
- One sector and a small company universe limit generalization.
- Event-text coverage is uneven, and Capital IQ Key Developments is not all financial news.
- Changing horizons, model selection, and preprocessing conventions complicate comparisons across archived runs.
- A 2024 holdout that was repeatedly inspected during development cannot support an untouched-test claim for later decisions.
- Optional models, statistical tests, and explainability plots must be distinguished from completed outputs.
- Signal simulations omit costs and realistic execution mechanics. Neither report establishes deployable trading performance.
06 / Read the original
The project report.
Sentiment-Augmented Financial Prediction: A Data Mining Approach to Market Signal Extraction
Max Bassler & Daniel Draper · dated May 8, 2026
Original report, unchanged. PDF copy provided for easy reading.
Evidence / Sources and version notes
What this page is based on.
Report evidence and the horizon discrepancy
The original project report supports the research question, team attribution, data sources (III.A), ratio rebuild and coverage (III.C), and selected-model ladder (IV.A.2). A companion report synthesis supplies the additional stability discussion in section 9. Its prose is not treated as independently attributable writing by Max.
The supplied report describes a 21-day target. The preserved final quarterly configuration (opens in a new tab) and the April migration conversation identify a 63-day sector-relative sign target. The numbers are presented as reported archived results, with that discrepancy retained. The model-training data and original predictions have not been rerun for this website.
The original report's claims about a five-model cohort, Bayesian tuning, and planned statistical analyses exceed what its completed results establish. They are not presented here as verified accomplishments.
Conversation coverage and attribution
The review retrieved actual messages from 37 conversations listed in the original Data Mining project, spanning March–May 2026, and the local Codex conversation Define branch contract covering the May 28–29 rebuild. The timeline identifies its specific conversations. Short quotations are Max's project messages; surrounding explanations are paraphrases. Pasted teammate correspondence and AI-generated prose were not treated as Max's original writing.
All seven available ChatGPT archive listing pages and the available Codex archive listing were checked; no clearly matching archived project title was found there. That does not prove no other history exists. Older messages or attachments not exposed by the available tools could not be verified, and summaries were used for discovery only.
Code, saved diagnostics, and reproduction limits
The public code archive (opens in a new tab) matches the project's layered pipeline and preserves earlier branches under variants/. Its final root snapshot is the later rebuild, not the original course run. The branch snapshot list (opens in a new tab) identifies those versions.
The later results were transcribed from the local ablation_report.md and signal_simulation_report.md read during this review. The original local project folder subsequently became unavailable. The public archive intentionally excludes datasets, predictions, trained models, and generated outputs, so a fresh end-to-end reproduction was not available. The saved diagnostics support the displayed values; this website does not certify the original runs.
No raw Capital IQ exports, full transcripts, private chat links, credentials, or unrelated personal details are distributed with this page.