Data

Data#

Services to check the quality of your data:

  • Azure Purview

  • Collibra

  • Profisee

  • Apache Griffin

  • Apache Atlas

  • admunsen

Data Storage

  • PostgreSQL


TileDB#

Data Economics

  • Production: How the data is produced, and stored?

  • Distribution: Who has access to the data? and by which channel?

  • Consumption: How to consume the data (i.e., computation).

    • Usually the focus

    • But usually domain-specific

Solution:

  • Data in a universal analysis-ready format

    • No ETL, no copies

    • Unified governance

    • Built-in marketplace

  • Universal data management platform

    • One infrastructure, any backend, any scale

    • Common for all data app

TileDB is likely be your solution

  • MultiDimensional arrays

  • secure governance & collaboration

  • Scalable, serverless compute

  • ta and code sharing and monetization

  • Pay-as-you-go, consumer pays

Submodular Optimization for Minimizing Redundancy in Massive DataSets#

Increases in size mean increases in redundancy:

  • Duplicates

    • in training sets can waste computational resources

    • between training and test sets can cause overly optimistic estimates of performance

\[ f(X \cup \{v\}) - f(X) \ge f(Y \cup \{v\}) - f(Y) \]

where \(X \in Y\) and \(v \notin Y\)

  • Diminishing returns property: the gain from adding in some specific element \(v\) to \(X\) decreases, or stays the same, as other elements are added to \(X\)

  • These functions don’t require continuous or differentiable

  • It’s been shown that greedy algorithm will find a subset within a constant factor of \(1-e^{-1}\) of the optimal subset, and empirical results show that the subset if almost always much closer

Feature-based Function#

A simple class of submodular function are feature-based functions

\[ f(X) = \sum_{d=1}^D \phi ( \sum_{x \in X} x_d) \]

where

  • \(\mathbf{X}\) is the set of selected examples,

  • \(x\) is an individual example

  • \(\mathbf{D}\) is the number of dimensions

  • \(d\) is the index of a particular dimension

  • \(\phi\) is a concave function (e..g, sqrt or log)

Graph-based Function#

Facility location is another submodular function, and has been used to specify the location of new facilitate

\[ f(X) = \sum_{y \in Y} \max_{x\in X} \phi (x,y) \]

where

  • \(\mathbf{Y}\) is the full set of items

  • \(\mathbf{X}\) is the set of selected items,

  • \(\phi\) is a similarity function that returns the similarity between 2 examples (i.e., a graph)

  • \(y, x\) are individual examples

Summary#

It’s important to choose the right sumodular function

  1. [Feature-based functions][Feature-based Function]hen each feature is a “quality of the data,” they work well. A higher value means more of that quality in the data (i.e., word counts work well, pixel values do not)

  2. [Graph-based Function] (e.g., facility location)can be used in a wide range of situations, but they need quadratic memory to store the similarity matrix, which takes up a lot of space.

apricot plays nice with PyTorch or TensorFlow