Formal Concept Analysis Research Toolbox

Formal Concept Analysis Research Toolbox (FCART) is intended for support of research in the field of Data Science: Classification, Clusterization, Attribute Exploration and other Data Analysis tasks. A new system should provide:

1. Universal integrated environment for knowledge and data engineers.
2. Built-in set of universal research tools based on FCA and multimodal clustering techniques for working with object-attribute data representation.
3. Additional tools for import/export of data and data preprocessing.
4. Extendibility of the research tools on several levels: from internal scripts to plugins.
5. Generation of rich, visually appealing reports.

Motivation

Our goal is to create a universal integrated environment for knowledge and data engineers. FCART is constructed upon an iterative data analysis methodology and provides a built-in set of research tools based on universal Formal Concept Analysis techniques for working with object-attribute data representations.

Main principles

1. Iterative process of data analysis using FCA entities and methods.
2. Separation of processes of data querying (from various data sources), data preprocessing (of locally saved immutable snapshots), data analyzing (in interactive visualizers of immutable analytic artifacts), and results formalizing (in a report editor).
3. Explicit definition of analytic artifacts and their types. It allows checking integrity of the session data and provides links between artifacts for an end-user.
4. Integrated performance estimation tools.
5. Integrated documentation of software tools and data analysis methods.

Main entities

Artifacts

Some of FCA entities appear to be fundamental to information representation. In FCART, we use the term “analytic artifact” which denotes the definition of abstract interface, describing the entity of the analytic process.

The basic artifact for FCA-based methods is that of “formal context”, i.e., object-attribute representation of a subject domain. Most important artifacts include “concept lattice” and “formal concept”.

All artifact instances are linked by “origination”. For example, we can generate the concept lattice from the formal context. In this case, the formal concept will be an "origin artifact" for the lattice. Another example is lattice and “association rules” – the lattice is the origin of the rules. Any artifact instance is immutable. It means that an instance cannot be changed after creation, but can be visualized in various ways.   

If we have the predefined set of artifacts in most cases we can use the term “artifact” instead of “artifact instance” without ambiguity. Collection of all artifacts in current analytic cycle forms so-called analytical session.

Solvers

All types of artifacts are generated by solvers. Each solver requires one or many artifact instances of preassigned types as input and produces one artifact instance of preassigned type as output.

Having predefined types of artifacts and links (assigned by solvers) between immutable artifact instances, we can check an integrity of data of particular analytical session. Without explicit user action, a session cannot lose any artifact instances and links, and guarantees integrity of a session.

Visualizers

Artifact visualizer is a special solver that generates user-oriented visual representation of input artifact instance. From a technical point of view, visualizer produces interactive or non-interactive window with some elements of user interface. Of course, one artifact can have different kinds of visual appearance.

Usually, visualizer is the last in a chain of solvers. However, we can get a visual representation of each artifact in a session.  For example, lattice browser generates a diagram of a lattice and allows a user to manipulate the diagram, but this browser does not generate new artifacts. We need to distinguish generation of new artifact and drawing of existing artifact for various purposes: working in the batch mode, increasing efficiency of long chains of solvers, benchmarking, etc.

Reports

Report is a final result of research. Every scientific environment must provide a report rich text editor with additional functionality to avoid mistakes while converting and moving multiple results with metadata to an external editor. The main feature of the editor is an automatic insertion of fully decorated artifact representation in the resulting report.

Technology

At this moment, we introduce the version of FCART in the form of local Windows application.

We use Microsoft and Embarcadero programming environments and different programming languages (C++, C#, Delphi, Python and other). For scripting we use Delphi Web Script and Python. Native executable (the core of the system) is compatible with Microsoft Windows 2000 and later and has not any additional dependences.

Another line of development is Web-version of system based on Microsoft .NET platform. For now architecture and some key components are ready, but we are going to focus on Web-development after finishing local version 0.9.

Main features

1. Querying external data sources.
2. Manipulating collections (with metadata) in LocalDB.
3. Generating snapshots as a main artifacts in form of many-valued contextes.
4. Effective building of concept lattice for binary context. 

Architecture

Current version of FCART consists of the following components.

• Core component includes:

1. multiple-document user interface of research environment with session manager and extensions manager,
2. snapshot profiles editor (SHPE),
3. snapshot query editor (SHQE),
4. query rules database (RDB),
5. session database (SDB),
6. main part of report builder.

• Local Data Storage (LDS) for preprocessed data.
• Internal solvers and visualizers.
• Additional plugins, scripts and report templates.

Main FCA workflow

From the analyst point of view, basic FCA workflow in FCART has four stages. On each stage, a user has the ability to import/export every artifact or add it to report.

1. Filling Local Data Storage (LDS) of FCART from various external SQL, XML or JSON-like data sources (querying external source described by External Data Query Description - EDQD). EDQD can be produced by some External Data Browser.
2. Loading a data snapshot from local storage into current analytic session (snap-shot described by Snapshot Profile). Data snapshot is a data table with anno-tated structured and text attributes, loaded in the system by accessing LDS.
3. Transforming the snapshot to a binary context (transformation described by Scaling Query).
4. Building and visualizing formal concept lattice and other artifacts based on the binary context in a scope of analytic session.

Screenshots

Roadmap

Links

1. Demo of FCA functionality in local version (0.8)