Ingestion is the process of acquiring unstructured text data from a variety of sources in order to be able to perform further analysis on it. This may involve, for example, taking text from social media posts, news articles, or customer service transcripts and putting it into a format that can be processed by software.
It can also refer more generally to the act of taking something in, such as food or information. In this sense, it is similar to the term input. However, while input may refer to any kind of data or information that is fed into a system, ingestion specifically refers to the process of acquiring and preparing text data for analysis.
Types of data ingestion
There are two main types of data ingestion: manual and automatic. Manual data ingestion is when a human manually enters the text data into the system. Automatic data ingestion is when the system automatically acquires the text data from an external source, such as through an API or web scraping.
There are several ways to ingest data:
1. APIs: An application programming interface (API) allows two pieces of software to communicate with each other. In the context of data ingestion, an API can be used to request and receive text data from an external source. For example, the Twitter API can be used to ingest tweets from Twitter.
2. Web scraping: Web scraping is a process of extracting data from websites. In the context of data ingestion, web scraping can be used to acquire text data from sources that do not have an API. For example, one could use web scraping to ingest reviews from a website that does not have an API.
3. Manual entry: As mentioned above, manual entry is when a human manually enters the text data into the system. This is generally not recommended for large amounts of data, but can be used for small datasets.
Comparison with similar terms
Ingestion is often confused with other similar terms, such as parsing and extraction. However, there are some key differences between these terms:
1. Ingestion is the process of acquiring unstructured text data from a variety of sources.
2. Parsing is the process of converting unstructured text data into a structured format.
3. Extraction is the process of extracting specific information from unstructured text data.