Mass Ingestion Framework
Comments
Nov 26, 2020
Atlassian uses cookies to improve your browsing experience, perform analytics and research, and conduct advertising. Accept all cookies to indicate that you agree to our use of cookies on your device.
Atlassian uses cookies to improve your browsing experience, perform analytics and research, and conduct advertising. Accept all cookies to indicate that you agree to our use of cookies on your device. Atlassian cookies and tracking notice, (opens new window)
Mass ingestion is the ingestion or replication of large amounts of data for use or storage in a database or a repository or files. The database can be a data lake, a cloud repository, or a Hadoop cluster. In addition it also provides support incremental/delta changes between batch loads, propagate schema changes/drift to the targets with support for partitioning/bucketing the data based on incoming data sets column definitions. Mass Ingestion service to ingest data at scale from selected database and streaming data sources and transfer the data with low latency to selected messaging systems and cloud targets.
Mass Ingestion provides the following ingestion solutions:
Mass Ingestion Databases: Propagates data from source objects in a database management system (DBMS) to multiple types of targets. A database ingestion
task can transfer a point-in-time snapshot of all source data in a schema to a target in a batch operation. A database ingestion task can also incrementally propagate DML data changes in near real time from an Oracle or SQL Server source to a target on a continuous basis. If you select the combined initial and incremental load type, the database ingestion task performs an initial load and then automatically switches to incremental load processing of data changes.
Mass Ingestion Files: Transfers a large number of files of different types between on-premises and cloud repositories. You can use Mass Ingestion Files to track and monitor file transfers.
Mass Ingestion Streaming: Transfers real-time streaming and machine data from selected sources to selected messaging systems and batch targets.
It aimed read database schemas, tables, columns based on quick metadata load from source database then propagate the results quickly to target stores. Target stores cloud anything like Cloud Object stores like GCS, ADLS, S3 and could any Database/Datawarehouse targets.