Dropdown

OCEAN-AI is an open-source framework for Big Five personality traits assessment and HR-processes automatization.

The models used in OCEAN-AI were trained on 15-second clips from the ChaLearn First Impression v2 dataset. For more reliable predictions, 15-second videos are recommended, but OCEAN-AI can analyze videos of any length. Due to limited computational resources on HuggingFace, we provide six 3-second videos as examples.

Step 1: Calculation of personality traits scores

Video Files (mp4, mov, avi, flv)

Radio

Files Web

Video Player

Select the video(s)

1	2	3

Export Big Five personality traits to a CSV file

Step 2: Solving practical task

Practical tasks,Практические задачи

Choose a practical task

Ranking potential candidates by professional responsibilities Forming effective work teams Predicting consumer preferences for industrial goods

Practical subtasks

Choose a practical subtask

16 Personality Types of MBTI Professional groups Professional skills

Potential candidates by Personality Type of MBTI (16)

What personality type indicator are you interested in?

Polarity traits threshold

Set value from 0 to 1.0

Polarity traits threshold

Set value from 0 to 1.0

Professional skills (4)

What professional skill are you interested in?

Openness target score

Set value from 0 to 1.0

Conscientiousness target score

Set value from 0 to 1.0

Extraversion target score

Set value from 0 to 1.0

Agreeableness target score

Set value from 0 to 1.0

Non-Neuroticism target score

Set value from 0 to 1.0

Equal coefficient

Set value from 0 to 1.0

Priority number

Set value from 1 to 11

Importance traits number

Set value from 1 to 5

Polarity traits threshold

Set value from 0 to 1.0

Potential candidates by professional responsibilities (6)

What profession are you interested in?

Openness weight

Set value from 0 to 100

Conscientiousness weight

Set value from 0 to 100

Extraversion weight

Set value from 0 to 100

Agreeableness weight

Set value from 0 to 100

Non-Neuroticism weight

Set value from 0 to 100

1	2	3

Personality types of MBTI are based on four Personality Dimensions

Dimension description	Dimension
Whether we prefer to live in a more structured way (making decisions) or in a more spontaneous way (taking in information)	(E) Extraversion - Introversion (I)

Dimension description	Dimension
How we interact with the world and where we direct our energy	(E) Extraversion - Introversion (I)
The kind of information we naturally notice	(S) Sensing - Intuition (N)
How we make decisions	(T) Thinking - Feeling (F)
Whether we prefer to live in a more structured way (making decisions) or in a more spontaneous way (taking in information)	(J) Judging - Perceiving (P)

Export ranking professional skill results to a CSV file

Video Player

First name

Last name

Phone number

{

"Ranking potential candidates by professional responsibilities": "16 Personality Types of MBTI" ,

"Forming effective work teams": "Finding a suitable junior colleague" ,

"Predicting consumer preferences for industrial goods": "Car characteristics" ,

"Ранжирование потенциальных кандидатов по профессиональным обязанностям": "16 персональных типов личности MBTI" ,

"Формирование эффективных рабочих коллективов": "Поиск подходящего младшего коллеги" ,

"Прогнозирование потребительских предпочтений в отношении промышленных товаров": "Характеристики автомобиля"

}

In development

OCEAN-AI is an open-source library consisting of a set of algorithms for intellectual analysis of human behavior based on multimodal data for automatic personality traits (PT) assessment. The library evaluates five PT: Openness to experience, Conscientiousness, Extraversion, Agreeableness, Non-Neuroticism.

Pipeline

OCEAN-AI includes three main algorithms:

Audio Information Analysis Algorithm (AIA).
Video Information Analysis Algorithm (VIA).
Text Information Analysis Algorithm (TIA).
Multimodal Information Fusion Algorithm (MIF).

The AIA, VIA and TIA algorithms implement the functions of strong artificial intelligence (AI) in terms of complexing acoustic, visual and linguistic features built on different principles (hand-crafted and deep features), i.e. these algorithms implement the approaches of composite (hybrid) AI. The necessary pre-processing of audio, video and text information, the calculation of visual, acoustic and linguistic features and the output of predictions of personality traits based on them are carried out in the algorithms.

The MIF algorithm is a combination of three information analysis algorithms (AIA, VIA and TIA). This algorithm performs feature-level fusion obtained by the AIA, VIA and TIA algorithms.

In addition to the main task - unimodal and multimodal personality traits assessment, the features implemented in OCEAN-AI will allow researchers to solve other problems of analyzing human behavior, for example, affective state recognition.

The library solves practical tasks:

Ranking of potential candidates by professional responsibilities.
Predicting consumer preferences for industrial goods.
Forming effective work teams.

OCEAN-AI uses the latest open-source libraries for audio, video and text processing: librosa, openSMILE, openCV, mediapipe, transformers.

OCEAN-AI is written in the python programming language. Neural network models are implemented and trained using an open-source library code TensorFlow.

Research data

The OCEAN-AI library was tested on two corpora:

The publicly available and large-scale First Impressions V2 corpus.
On the first publicly available Russian-language Multimodal Personality Traits Assessment (MuPTA) corpus.

Publications

Journals

            
@article{ryumina24_prl,
    author = {Ryumina, Elena and Markitantov, Maxim and Ryumin, Dmitry and Karpov, Alexey},
    title = {Gated Siamese Fusion Network based on Multimodal Deep and Hand-Crafted Features for Personality Traits Assessment},
    journal = {Pattern Recognition Letters},
    volume = {185},
    pages = {45--51},
    year = {2024},
    doi = {https://doi.org/10.1016/j.patrec.2024.07.004},
}
@article{ryumina24_eswa,
    author = {Elena Ryumina and Maxim Markitantov and Dmitry Ryumin and Alexey Karpov},
    title = {OCEAN-AI Framework with EmoFormer Cross-Hemiface Attention Approach for Personality Traits Assessment},
    journal = {Expert Systems with Applications},
    volume = {239},
    pages = {122441},
    year = {2024},
    doi = {https://doi.org/10.1016/j.eswa.2023.122441},
}
@article{ryumina22_neurocomputing,
    author = {Elena Ryumina and Denis Dresvyanskiy and Alexey Karpov},
    title = {In Search of a Robust Facial Expressions Recognition Model: A Large-Scale Visual Cross-Corpus Study},
    journal = {Neurocomputing},
    volume = {514},
    pages = {435-450},
    year = {2022},
    doi = {https://doi.org/10.1016/j.neucom.2022.10.013},
}

Conferences

            
@inproceedings{ryumina24_interspeech,
    author = {Elena Ryumina and Dmitry Ryumin and and Alexey Karpov},
    title = {OCEAN-AI: Open Multimodal Framework for Personality Traits Assessment and HR-Processes Automatization},
    year = {2024},
    booktitle = {INTERSPEECH},
    pages = {3630--3631},
    doi = {https://www.isca-archive.org/interspeech_2024/ryumina24_interspeech.html#},
}
@inproceedings{ryumina23_interspeech,
    author = {Elena Ryumina and Dmitry Ryumin and Maxim Markitantov and Heysem Kaya and Alexey Karpov},
    title = {Multimodal Personality Traits Assessment (MuPTA) Corpus: The Impact of Spontaneous and Read Speech},
    year = {2023},
    booktitle = {INTERSPEECH},
    pages = {4049--4053},
    doi = {https://doi.org/10.21437/Interspeech.2023-1686},
}