Data science: processing, visualization and text analytics

What is data science?

Data science is an interdisciplinary field that uses computer science (including machine learning), statistics and other disciplines to extract information from Big Data, or from data in general. Because alternative data are often unstructured, data scientists have to treat their structure before analysis can start.

How are data processed?

Five methods cover the path from raw feed to analysis tool. Learn them in order and with one-line meanings.

Data processing

• Capture Memorize collect and format; latency
• Curation Memorize clean; fix errors and gaps
• Storage Memorize record, archive, access; database design
• Search Memorize query to locate content
• Transfer Memorize move to the analysis tool

• Data capture : how data are collected and put into a usable format. Latency is the delay on a network. Low-latency systems carry high volumes of data with minimal delay and are essential for automated trading that reacts to real-time prices. High-latency systems do not need real-time data.
• Data curation : ensuring quality and accuracy through cleaning. Review all data, detect bad or inaccurate records and adjust for missing data where appropriate.
• Data storage : how data are recorded, archived and accessed, including database design. A key point is whether data are structured or unstructured, and whether the analysis needs low latency.
• Search : how to query data. Big Data needs applications that can examine large quantities and locate requested content.
• Transfer : how data move from the source or storage location to the analytical tool, for instance through a direct feed from an exchange.

Step	Example
Capture	A millisecond price feed for an execution algorithm
Curation	Removing duplicate trades; flagging a price 100 times too large
Storage	Tables for prices, a document store for filings
Search	Finding every mention of a supplier across 10 years of filings
Transfer	An exchange’s direct feed into the model

Exam tip Clean, store, find, move. Curation is cleaning; storage is archiving and access.

Memorize

How are the results shown?

Data visualization is how data are formatted, displayed and summarized graphically.

Data visualization

• Structured data
  • Tables, charts, trend lines
• Big or unstructured data
  • Heat maps, tree diagrams, network graphs
  • Interactive 3D graphics color, size, shape add dimensions
• Text data Memorize
  • Tag cloud Memorize word frequency
  • Mind map Memorize links between concepts

Structured data suit tables, charts and trend lines. Unstructured or very large data need richer tools: heat maps (color shows value, such as a correlation matrix), tree diagrams, network graphs and interactive 3D graphics where users can zoom and rotate. With more than three variables, color, shape and size add dimensions.

For text, two tools matter. A tag cloud sizes words by how often they appear. A mind map varies the idea: instead of frequency, it shows how concepts are related.

What do text analytics and NLP do?

Text analytics uses computer programs to analyze and derive meaning from large unstructured text or voice datasets: filings, reports, earnings calls, social media, email and surveys. It includes automated retrieval, word-frequency analysis and pattern recognition, and it can feed predictive models that look for short-term indicators of coming trends, such as a shift in consumer sentiment.

Natural language processing (NLP) is an important application within text analytics. It draws on computer science, AI and linguistics to develop programs that interpret human language, in text and in speech. Tasks include translation, speech recognition, text mining, sentiment analysis and topic analysis.

Investment uses: scoring the tone of earnings calls and analyst commentary, which can reveal shifts before a recommendation changes; scanning thousands of news items for company events; watching wording in central bank statements; and helping compliance teams flag unusual communications or protect confidential information. Machines can do this at a scale and speed no analyst team can match.

Which languages and databases come up?

In the table, VBA is Visual Basic for Applications and SQL is Structured Query Language.

Tool	Use
Python	Free, easy to learn, basis of many fintech applications
R	Free, built for statistical analysis, ML and econometrics
Java	Runs on many systems; underlies many internet applications
C and C++	Optimized for speed; algorithmic and high-frequency trading
Excel VBA	Macros to automate spreadsheet tasks and data collection
SQL	Query language for structured tables, run on a server
SQLite	Embedded structured database, common in mobile apps
NoSQL	Database for unstructured data that do not fit rows and columns

Go deeper Latency and automated trading

A trading program that reacts to a price change gains nothing if its data arrive late. That is why capture and transfer for such systems are built for minimal delay. A model that reviews filings once a quarter loses nothing from a slower link, so it can use cheaper, higher-latency systems.

Go deeper Sentiment scores as inputs

Some quantitative strategies treat a sentiment score as one more input, next to valuation or momentum. If markets absorb tone slowly, the score has value; if prices already reflect it, it does not. That question belongs to market efficiency.