Big data, the four Vs and alternative data

What counts as financial data?

Financial decisions use two kinds of data. Quantitative data are numbers: prices, returns, interest and exchange rates, accounting figures and economic indicators. Qualitative data are not: market sentiment, management quality, environmental factors and geopolitical influences. Natural language processing (NLP) and sentiment analysis can turn qualitative data into numbers that sit beside the quantitative ones (the third lesson of this module covers both).

Three definitions set the scene.

• Big data in finance are the very large volumes of structured and unstructured data that markets produce (transactions, price moves, investor behavior), now joined by alternative data such as social media sentiment, satellite images and phone data.
• Financial data science is a specialized branch of data science (third lesson) that applies its principles to financial data: prices, rates and returns; accounting information and disclosures; and economic data. It supports investment, portfolio, trading and risk decisions, and it needs domain knowledge of markets, regulation and instruments.
• High-frequency data are recorded at extremely short intervals, often milliseconds or microseconds, such as trades and quotes. They are granular, huge in volume and useful only if processed at once, so they need low-latency systems.

Exam tip What makes it financial data science is the data and the context it is applied to. It is not limited to numbers (it uses qualitative data too) and it does not rely on machine learning alone.

What does fintech change for investors?

Fintech is technological innovation in the design and delivery of financial services and products. In everyday use the word also means the companies that build these tools, and their sector. It began with data processing and the automation of routine tasks, moved to systems that execute decisions by set rules, and now includes programs that learn from data. It has produced new systems for investment advice, financial planning, business lending and payments, and big data are a driver of it.

For the exam the focus is analysis: very large datasets, including new non-traditional data, and the artificial intelligence (AI) and machine learning (ML) tools that handle them.

What makes data big?

Big data are the vast, fast-growing datasets produced by markets, companies, governments, individuals and devices. Three features define them, and a fourth matters when you rely on them.

Big data

• Volume Memorize how much
• Velocity Memorize how fast it is created and delivered
• Variety Memorize how many forms
  • Structured tables, fixed fields
  • Semi-structured web pages, tagged news
  • Unstructured text, images, audio, video
• Veracity Memorize credibility and reliability, added for inference

V	Question it answers	Example
Volume	How much data?	Billions of card transactions a year
Velocity	How fast is it created and needed?	Order-book updates every millisecond
Variety	How many forms?	Prices, scanned filings, satellite photos, audio
Veracity	Can we trust it?	Social-media posts written by bots

Volume is the number of data points, now millions or billions. Volumes have grown from megabytes and gigabytes to terabytes (trillions of bytes) and petabytes (quadrillions). Velocity is the speed and frequency of recording and transmission, with real-time or near-real-time data now common. Variety is the mix of sources and formats: structured, semi-structured or unstructured. Veracity is the credibility and reliability of a source. It is the exam’s favorite confusion partner for variety: variety asks “what shape?”, veracity asks “is it true?”.

Exam tip Fake reviews, bot posts and unreliable or biased sources are veracity. Scanned filings, audio and images in one dataset are variety.

Where do big data in finance come from?

Source	Examples
Financial markets	prices and trades in shares, bonds, derivatives, commodities
Businesses	company accounts, card purchases, customer purchase histories
Governments	trade, employment, tax and economic releases
Individuals	reviews, searches, browsing, social posts
Sensors	satellite images, ship and aircraft tracking, traffic counts
Internet of things	smart buildings reporting energy use, climate control and security

Structured or not?

• Structured data fit rows and columns, with fields that allow comparison across records: prices, financial statements, Structured Query Language (SQL) tables, comma-separated values (CSV) files.
• Semi-structured data mix both. They do not fit a table, yet they can be tagged, stored and organized: web pages in HyperText Markup Language (HTML) code, news articles tagged by company and date.
• Unstructured data have no tabular form: social media, email, texts, voice recordings, photos, video. They need specialized programs before an analyst can use them.

Memorize

Where does alternative data come from?

Alternative data are information from non-traditional sources, such as social media activity, satellite images, web traffic, real-time retail sales and location data. They sit beside traditional data (market prices, financial statements, regulatory filings, economic releases). Investors use them to read sentiment on events as it forms, to track online prices over time and so estimate inflation and spending as they happen, to monitor supply chains for bottlenecks, and to assess companies’ environmental impact. A growing number of specialist firms collect, package and sell alternative datasets. There are three main sources.

Alternative data

• Individuals Memorize mostly unstructured
  • Social media, news, reviews
  • Web searches, clicks
• Business processes Memorize mostly structured
  • Transactions and sales data
  • Corporate operational data supply chain, banking records, scanner data
• Sensors Memorize huge volume
  • Satellites, cameras, phones
  • Internet of things connected devices and buildings

• Individuals produce text, photos, video and audio, and also clicks and time on a page. This is mostly unstructured and growing quickly as more people shop, post and search online.
• Business processes produce information flows from companies and public bodies: direct sales data such as card purchase histories, plus corporate operational data (supply-chain information, banking records, retail point-of-sale scanner data). It is mostly structured.
• Sensors are devices such as phones, cameras, radio-frequency identification (RFID) chips and satellites. Their data are mostly unstructured and many orders of magnitude larger than the other two. Networks of embedded devices, from smart buildings to vehicles, form the internet of things (IoT).

What makes big data hard to use?

Big data are not free insight. The data must be sourced, cleansed and organized before any analysis, and that is hardest for alternative data, because much of it is qualitative (text, photos, video), open to several readings and dependent on context. These big data challenges come down to three questions:

Big data challenges

• Quality Memorize selection bias, missing data, outliers
• Volume Memorize is there enough?
• Appropriateness Memorize does it suit the analysis?

• Quality: does the dataset have selection bias, missing data or outliers?
• Volume: is there enough of it?
• Appropriateness: does it suit the analysis?

Traditional methods cannot always cope with such data, which is why AI and ML (next lesson) are used alongside them. Legal and ethical limits apply too. Web scraping , the automated extraction of data from websites with software tools or scripts, can capture personal data that are protected by data-protection rules or were published without the person’s knowledge and consent. Best practices are still developing, and national regulators’ guidance can conflict.

Go deeper Why unstructured data dominate growth

Every phone is a camera, microphone and location sensor, and every connected appliance adds a stream. These devices create images, audio and readings, not neat tables. Structured data grow too, but far more slowly, so the share of data an analyst can use without specialized tools keeps shrinking. That is why the next lesson’s tools matter.

Go deeper Alternative data and the Standards

Data that are not public raise a question the Standards of Professional Conduct address: could the information be material and non-public? A dataset assembled from licensed, lawful sources is a research input. A feed that leaks a company’s unreleased results is not. Always ask how a vendor obtained the data and whether individuals consented to its use.