Human Interaction Motion Generation From Text

ShanghaiTech VRVC Center, Immersive 4D Group

Sep 2022 - Jan 2023

Undergraduate researcher, supervised by Prof. Lan Xu

I join in a research group under VRVC center called RhythMo, it is now also a digital technology Co. Our team proposed a novel approach for human interaction motion generation from text. There are many reasearches about human motion generation, such as CLIP, DDPM, but very few about human interaction generation, so we want to generate some interaction between two people in our daily life.

To achieve our goal, we invited many volunteers to participate in our data collection, including actors who recorded daily interactions, and many professional practitioners to record some technical movements such as street dance, tai chi, basketball, fencing, etc.

I took active part in the data collection and annotation. During the research, I was able to experience first-hand the research process, learn to use research equipment, manage data and perform data processing. It is my first time to participate in research so I may not have made a very big contribution to method improvement and design, but I still learned a lot about generation model such as CLIP, DDPM. All these methods are widely used in the field of generation nowadays.

Here are some photos about my experience in VRVC lab

In addition, I also made a little project: human motion generation, under the guidance of a PhD student, Han Liang. We proposed a novel approach for human motion generation from text. Our model takes as input a text description of a motion, and outputs a sequence of motion frames that represent the motion described in the text. To obtain a complete motion generation model, we combine the CLIP motion embedding decoder with an a prior model that generates possible CLIP motion embedding from a given text heading, and then this embedding is used to condition a diffusion decoder which produces a final motion.

You can learn more about this little project Human-motion generation from text.

Close

TSLink: an interactive AI-supported, bias-aware, adaptable interactive MOOC learning system

ShanghaiTech University, Viseer Lab

Undergraduate researcher, supervised by Prof. Quan Li

Introduction

Instructor-student communication in educational settings is profoundly influenced by the curse of knowledge, a cognitive bias that causes experts to underestimate the challenges faced by learners due to their own in-depth understanding of the subject. This bias can hinder effective knowledge transfer and pedagogical effectiveness. To address this issue, we introduce TSLink, a bias-aware, adaptable interactive MOOC (Massive Open Online Course) learning system.

Formatvie Study

To address our research questions, we conducted a survey involving 192 students from various academic backgrounds and degrees, complemented by expert interviews with 7 instructors across different disciplines at a local university. This study aims to mitigate the bias introduced by the curse of knowledge in the current teaching process, with the goal of improving the teaching experience for both instructors and students and fostering greater alignment between them. To achieve this, we conducted an survey with students and a series of semi-structured interviews with instructors. Here is the brief summary of the results.

• Finding 1: The Necessity of instructors’ proactive assessment of learning status.
• Finding 2: Learning challenges affect the willingness to communicate with instructors.
• Finding 3: Expertise in recognizing student understanding.
• Finding 4: Ensuring majority comprehension within teaching constraints.
• Finding 5: The impact of prerequisite knowledge on communication.
• Finding 6: Embracing online platforms for enhanced learning.

System Design

Based on the six key findings, our work aims to integrate AI methods and visualization strategies into online education platform interfaces tailored for students and instructors. This integration aims to create a more effective learning environment and feedback loop, mitigating the impact of curse of knowledge bias on both groups. The student end is designed to provide systematic learning guidance and encourage more granular feedback, while the instructor end is designed for comprehensive and nuanced analysis of that feedback.

Student End

TSLink Student System Page Display: (A) Teaching video playback area. (B) Video-related functional sub-areas, specifically including chapter selection area, comment area, and course completion selection area. (C) Knowledge network visualization interactive area, where students can interact with the network graph using the mouse. (D) Knowledge point information display area, where students can click on relevant knowledge points in the network graph to view detailed information about each point and perform self-assessment scoring.

Here is the detailed demonstration of student network, if you are interested, please refer to my thesis paper.

TSLink Teacher System Page Display: (A) Teaching video playback area. (B) Student comment overview, with options to sort by different criteria. (C) Visualization area for student video playback data, with interactive images. (D) Knowledge network image display area, showing a summary of students’ scoring data.

Here is the detailed demonstration of teacher end interaction, if you are interested, please refer to my thesis paper.

Close Project

Real Time Coding Editor

Motivation

In classroom environments, students are frequently organized into groups to support collaboration. This is typical in programming classes for small group projects, peer reviews, or discussions on coding problems. However, there are various criteria for grouping students, such as coding performance, code structure, or previous group interactions. We need to comprehend the interplay between these dynamics and the outcomes of the learning sessions. Additionally, we must explore how to support these groups to enhance the effectiveness of their discussions, making them more meaningful and fulfilling for the students.

Problem Solving

Through reading relevant papers about real time coding design, I learnt that task distribution and efficient discussion are two key problems students may have during team working process. Comparing with traditional VCS system, students prefer to use real time editor to cooperate with others.

My design focuses on two points: making task visualization and distribution more efficient and connecting discussion with the real time editor or the specific code. Here is my first draft.

My idea is using annotation attached to some lines of code blocks to connect comments with specific code. However if the source code has been modified, this design will cause a lot of mess and if there are thousands of comments on this code file, how to arrange such interface can be a big problem.

So I improve my idea using a VCS manage system to solve this mess, using a VCS system to manage the discussion and code versions. And I make my task distribution idea more specifically.

However, my sketches will hit scalability issue. imagine if this is a classroom setting where the instructor has the main codebase and they invite 100 students to participate in writing a specific function. This tiny interface can not manage such situation. After a long time of brainstorming and reading relevant materials, I finally come with the idea that can perfectly handle such problems and I beautify the whole interfaces to make it more simple and easy-understanding.

Final Prototype

Overview

To solve the problem that my previous idea could not afford too many people to edit and discuss the same code block, I decided to make the discussion board to be a new page instead of a chat box attached to the code. But the main editor is still a real-time editor that can allow multiple users to edit together.

For this image, on the top is the main editor and you can select the codes and make it to be a discussion issue block, which means you have problems or something to chat with your teammates. so you lock this area and others can not edit it until this issue is solved. The red block on the editor means there is a discussion issue related to this code and you can click the red block to access to the discussion detailed page.

Discussion Board Overview

This is a new page that shows the discussion board, and it shows all the discussion issues related to the editor, you can also add an issue at this page by selecting the relative code lines.

This one is the detailed page where when you click one of the issues, you can see the detailed discussions, the original problems and replies, each one is bound to one of the code versions shown below. The code block on the right side is related to the real-time editor. However, it is not a real-time editor, you can click the code verison in the version list to see different code versions provided by different people related to this discussion, or add a new version and edit yourself. Every code version can only be edited by one person who generates it, and if you are not finishing your code, it will not be shown in the list. The original version is the code selected by the issue owner. And if other members have other ideas that want to help him, he can edit the code from any existing versions in the version list. To test different versions in the main editor, I design a button on the version list to select which version you want to be shown in the main real-time editor, and then you can test this version in your project. Only one code version will be shown in the main editor.

In the above version 1, I design all the code versions to be flat, and if you want to edit based on others code version just click the Edit button and it will generate a new version for you to edit. So if 100 people are participating in this discussion and they all want to change the code, there will be 100 new versions and they can edit separately.

In this version all the code versions are connected and can be shown by a version graph. When you try to edit the code and click the Edit button, the VCS system will generate a new version using a hash number as its commit tag, and this new version will be a node added on the version graph. Also when you do not click the upload button, others will not see your version in the list. When the discussion is done, the issue owner can close the discussion and select one of the code versions in the list to be shown in the main editor. And then this code area will be unlocked, allowing users to edit. This "lock" just means that other users in the real-time editor can not edit it, otherwise it would be conflicts.

Task Distribution

The task distribution part is not about solving the scale-up problem, I just refresh it to be more nice-looking and easier to show your project status.

Close Project

Simple Website Design

University of Wisconsin-Madison, Building User Interfaces (CS571)

Here are some of my courework related to simple website design, using React to construct a dynamic website.

Here is my first website design work using React, Display a simple website showing some snacks and their relevant information.

Here is my first dynamic webstie design, displaying a simple cat adoption website, you can choose buds interest you and click show more to get more information. Then clicking save to add them into your basket. In the basket page, you can also choose to finally adopt them or unselect them.

Here is a simple chatroom design, you can register your own account to login into the chatrooms, you can post your chat and delete what you post when you are logged in. However, when you visit the chatroom as a guest, you can only view the chat messages while you can not post or delete any post.

Close Project

Simple Apps Design

University of Wisconsin-Madison, Building User Interfaces (CS571)

Here are some of my courework related to simple app design, using React Native to construct a dynamic app.

Here is my first app design work using React Native, Display a simple app showing some snacks and their relevant information. You can add numbers to calculate your order.

Here is a news app that shows the current wisc news. You can go the every detailed page of every article and also go the preference page to turn on or turn off some tags to show the articles you interest.

Here is a chatroom app that uses the same database as the chatroom website. So the basic functions are the same as the website, register, login, logout, enter as a guest, post, delete and view the chatrooms.

Close Project

Figma Prototype Design

These are figma design that I learnt by myself in my spare time.

Here is a simple food delivery app that focuses on Chinese food, showing simple get started page and detailed restaurants information.

Here is the bakery app figma design, based on my previous app design. I beautify the whole interfaces and add more interactions.

Close Project

Chrome Dino Game

ShanghaiTech University, Computer Architecture I (CS110)

Chrome Dino pixel game on Longan Nano development board.

In this project, we implemented a Chrome Dino Runner Game using the Longan Nano development board. We utilized PlatformIO to interact with the development board and support cross-language programming. For example, we implemented most of the framework by C, but some of the auxiliary functions are implemented by RISC-V assembly. The functions can be invoked bidirectionally.

The game we implemented features:

• A simple startup menu and diffuculty selection page.
• A score board.
• Randomly generated cactus and pterosaurs acting like obstacles.
• The Dino can jump or creep controlled by the 2 buttons.
• Super mode in which the dino can hit everything without failing the game.

Human Motion Generation From Text

ShanghaiTech University, VRVC lab, RhythMo

Introduction

Motion generation from text is a recently emerging field of research. It involves the task of generating a sequence of frames representing a human motion from a text description. This can be used for applications such as virtual reality, animation, and video games. The task is challenging as it requires understanding of natural language, as well as the ability to generate realistic motion.

In this work, we combine these two approaches to solve the problem of text conditional motion generation. We first train a diffusion decoder to invert the CLIP encoder. Our inverter is non-deterministic and can generate multiple motions corresponding to a given motion embedding. The presence of the encoder and its approximate inverse (decoder) allows for the ability to go beyond text-to-motion translation.

Here is our pipline:

It's my first time to learn and do some research in deep learning so I can't write the whole method by myself actually the resource code is not written by my own. But I actually take an active part in this research and I speed a lot of time learning those essays about motion generation and try to understand the whole process. The process means a lot to me. Also, I help them annot many data and record the training data, that's the most I can do.

I really learned a lot from this wonderful research experience and I spend a lot of effort on it. Also I can't do much about method programming, I still learned so much from the whole process and the other lab members.

A Space Efficient Algorithm for LCSK

ShanghaiTech University, Algorithm Design And Analysis (CS240)

The LCS problem is the problem of finding the largest sequence common to all sequences in a set (usually only two sequences). It has many applications such as pattern matching, file comparison and so on. The article we reseached proposed two space efficient algorithms for LCSk and LCS≥k problems based on algorithms of O(kn) space complexity, if the size of given sequences are both n.

Close Project

It is an unfinished hand-made animation video making by my own. I made it for my favourite character in this comic.It is very short now, maybe one day I can finish it.

Here is the video :)

Close

RV32I-RVC Bidirectional Translator

ShanghaiTech University, Computer Architecture I, (CS110)

It is a project about writing an translator that translates RISC-V instructions to RISC-V compressed instructions for the first part. And for second part, we implement a translator that converts 16-bit RISC-V Compressed (RVC) instructions to equivalent 32-bit RISC-V instructions.

Here is the website introducing this project in detail: First part of project, Second part of project

Basically, it is not a hard job to understand but it takes our a long time to implement it. For me, I wrote 1000 lines code to finish the first part of this project, it really a huge challenge for me to write and debug such long and complicated code. I even made a specific form about it, it really took efforts.

Though the process is very challenging, I learn a lot from this precious experience, writing a huge project with so many lines code and finally succeeded on my own.

Close Project

Art Work

Here is my first drawing using canvas, I made a picture of dva, who is a character in Overwatch.

Here is my first drawing using SVG, she is also a character from Overwatch, she is my favourite hero Kiriko.

Here is my drawing, another version of dva.

Close

JS Animation Art work

Warning: They are works I complished in my Computer Graphics course so please DO NOT copy or steal some codes from this website. They are just for demonstration.

Animation Work

Here is my first animation using JS, a beautiful fireworks made by myself. Just click to add more fireworks. :D It will generate some random fireworks too.

Here is an animation about flocking style, I implement bounce off, alignment, cohesion and mouse attraction moving style.

Here is a train animation work, you can drag the points on the canvas to change the railway, press shift and click to add new control point, press ctrl and click the points existed to delete point. Select the simple track or complex track, using art length parameterization or not. Also, try the tension slider to see a difference!

Close

Blizzard World Details

Thank you for being interested in this special project! It really took me a lot of efforts to complete it all by myself, the source code is nearly 6000 lines. I just want to show some images I screenshoted myself for this project in case you can not find these details in the dynamic model. And also some drafts I made during my progress, maybe you do not notice some complex objects in my theme, it is really hard to create a hollow building using Three.js. :)

Here is a sreenshot of the Blizzard World theme in the Overwatch, using this screenshot to primarily construct my theme.

Here is the initial draft I made when I just began to construct the whole theme, using a top-down view to draw the basis.

Here is one of the most difficult objects I build in my theme, this roon is hollow inside and I build little horses in this house that can lead you to view the inside.

Here is another hollow objects that really requires a lot of efforts to make it, I made many details to remake the original one.

Here is a screenshot when I finished most of my construction, however drawing the ground texture and making travelguide routines needs accurate calculations. So I screenshot it to calculate myself.

These are some screenshot I photoed myself to help you enjoy the beauty of my theme!

Here are some pictures of the scene I made myself compared to the original game.

Close Project

My CG Final Project: Blizzard World

Warning: They are works I complished in my Computer Graphics course so please DO NOT copy or steal some codes from this website. They are just for demonstration.

My town's theme is a theme Parks, Blizzard World, also a attack escort map in video game Overwatch. It is my favourite game and Blizzard World is a theme park map, which fits my previous works, I can place some parkobjects in this theme.

I did my best to remake the original theme in game, making many details to make it more like the Blizzard World in Overwatch. I also make some modifications to make it more beautiful. If you have played Overwatch, hope you can easily recognize it!

Click Run to run the world, use slider to control the speed of whole world.

Use mouse to control your view when you use Obrit Camera, you can follow/drive some objects using Drive/Follow Objects. For Flying Camera, use W/A/S/D/R/Q to control your view.

If you are interested in my process of making such huge theme, you can go to the Detailed Page

Close