Data science: processing, visualization and text analytics

What is data science?

Data science is an interdisciplinary field that combines statistical methods, mathematical approaches, computational tools and domain knowledge to handle and interpret big data. It covers the whole pipeline, from collecting and cleaning data to modeling and interpreting it. Because alternative data are often unstructured, data scientists have to treat their structure before analysis can start.

Large datasets need infrastructure. A data lake is a central store that keeps structured, semi-structured and unstructured data in their original format. A distributed computing framework is software that splits processing across many machines. For very large datasets, artificial intelligence (AI) techniques may find complex, non-linear relationships that traditional statistics miss.

What makes financial data hard to model?

Financial data have traits that shape the methods used. These characteristics of financial data are:

What makes financial data hard

• Volume and speed Use it vast, very fast
• Time dependence Use it order matters
• Noise and non-stationarity Use it patterns hide and shift
• Non-linear, changing dependence Use it copulas, neural networks
• Seasonal and cyclic patterns Use it repeat on a calendar
• Fat tails Use it extremes more frequent
• Sparse or missing data Use it short or patchy histories

• High volume and speed: markets produce vast amounts of data very fast.
• Time dependence: observations come in order and depend on earlier ones, so time-series methods are needed.
• Noise and non-stationarity: random errors and irrelevant data hide patterns, and with non-stationarity the patterns themselves shift over time.
• Non-linearity and changing dependence: relationships between assets are often non-linear and change. A copula is a function that joins each variable’s own distribution into a joint distribution, capturing how the variables depend on each other. Copulas and neural networks can model such links.
• Seasonal and cyclic patterns: quarterly earnings seasons, swings between low-rate and high-rate periods.
• Fat tails: extreme moves happen more often than a normal distribution predicts, so a model built on the normal distribution underestimates risk (see statistical distributions). These are fat tails .
• Sparse or missing data: short or patchy histories, for example in some emerging markets. Mishandled gaps bias results.

How are data processed?

Five methods cover the path from raw feed to analysis tool. Learn them in order and with one-line meanings.

Data processing

• Capture Memorize collect and format; latency
• Curation Memorize clean; fix errors and gaps
• Storage Memorize record, archive, access; database design
• Search Memorize query to locate content
• Transfer Memorize move to the analysis tool

• Data capture : how data are collected and put into a usable format. Latency is the delay on a network. Low-latency systems carry high volumes of data with minimal delay and are essential for automated trading that reacts to real-time prices. High-latency systems do not need real-time data.
• Data curation : ensuring quality and accuracy through cleaning. Review all data, detect bad or inaccurate records and adjust for missing data where appropriate.
• Data storage : how data are recorded, archived and accessed, including database design. A key point is whether data are structured or unstructured, and whether the analysis needs low latency.
• Search : how to query data. Big data need applications that can examine large quantities and locate requested content.
• Data transfer : how data move from the source or storage location to the analytical tool, for instance through a direct feed from an exchange.

Step	Example
Capture	A millisecond price feed for an execution algorithm
Curation	Removing duplicate trades; flagging a price 100 times too large
Storage	Tables for prices, a document store for filings
Search	Finding every mention of a supplier across 10 years of filings
Transfer	An exchange’s direct feed into the model

Exam tip Clean, store, find, move. Curation is cleaning; storage is archiving and access.

Memorize

How are the results shown?

Data visualization is how data are formatted, displayed and summarized graphically.

Data visualization

• Structured data
  • Tables, charts, trend lines
• Big or unstructured data
  • Heat maps, tree diagrams, network graphs
  • Interactive 3D graphics color, size, shape add dimensions
• Text data Memorize
  • Tag cloud Memorize word frequency; also word cloud
  • Mind map Memorize links between concepts

Structured data suit tables, charts and trend lines. Unstructured or very large data need richer tools: heat map s (color shows value, such as a correlation matrix), tree diagrams, network graphs and interactive three-dimensional (3D) graphics that users can zoom and rotate. With more than three variables, color, shape and size add dimensions.

For text, two tools matter. A tag cloud (also called a word cloud) sizes words by how often they appear. A mind map varies the idea: instead of frequency, it shows how concepts are related.

What do text analytics and NLP do?

Text analytics uses computer programs to analyze and derive meaning from large unstructured text or voice datasets: filings, reports, earnings calls, social media, email and surveys. It includes automated retrieval, lexical analysis (analysis of word frequency in a document) and pattern recognition on key words and phrases. It can feed predictive models that look for indicators of future performance, such as consumer sentiment, and it works on private as well as public information (customer feedback, internal email).

Natural language processing (NLP) is an important application within text analytics. It draws on computer science, AI and linguistics to develop programs that interpret human language, in text and in speech. Tasks include translation, speech recognition, text mining, sentiment analysis and topic analysis.

Investment uses: scoring the tone of earnings calls and analyst commentary, which can reveal shifts before a recommendation changes; scanning thousands of news items for company events; watching wording in central bank statements; and helping compliance teams flag unusual communications or protect confidential information.

NLP is built for unstructured text and speech, not for structured tables or price time series. It does not replace the analyst: it reads at a scale and speed no team can, and the analyst judges what it finds.

Central bank statements and speeches are a natural NLP input, because officials signal through their choice of topics, words and tone. NLP can track rising or fading themes such as inflation or growth.

Which languages and databases come up?

In the table, VBA is Visual Basic for Applications and SQL is Structured Query Language.

Tool	Use
Python	Free, easy to learn, basis of many fintech applications
R	Free, built for statistical analysis, machine learning and econometrics
Java	Runs on many systems; underlies many internet applications
C and C++	Optimized for speed; algorithmic and high-frequency trading
Excel VBA	Macros to automate spreadsheet tasks and data collection
SQL	Query language for structured tables, run on a server
SQLite	Embedded structured database, common in mobile apps
NoSQL	Database for unstructured data that do not fit rows and columns

Go deeper Latency and automated trading

A trading program that reacts to a price change gains nothing if its data arrive late. That is why capture and transfer for such systems are built for minimal delay. A model that reviews filings once a quarter loses nothing from a slower link, so it can use cheaper, higher-latency systems.

Go deeper Sentiment scores as inputs

Some quantitative strategies treat a sentiment score as one more input, next to valuation or momentum. Like any input, it earns its place only if it still helps predict returns on data the model has not seen, which is why the test set from the machine learning lesson matters here too.