News Blog
This page contains news items and blog posts about activities at the lab.
Kelsey's Blog Post
Kelsey is a high school student who participated in a summer research experience at the lab during June-August 2009. She recorded a video blog entry about her experience, and she also used software at the lab to create an ASL animation expressing the same information as her video. She has also prepared an English version of her blog post which is included below.
English Version
Hi! My name is Kelsey. I am deaf, and my family is hearing. I live in Long Island. I attended the Mill Neck School for the Deaf from when I was 13 months old until I was 10 years old. I transferred to James H. Vernon (middle school) then to Oyster Bay High School (OBHS). I will be a senior this fall, and I participate cheerleading as sport. I came to Queen College for an internship. The internship is for an ASL project to try to improve ASL animations. I am intern here for 8 weeks.
We use many different programs here. One program that we use is called Gesture Builder 2.0; this is a program that we use to make new animations of signs. Gesture Builder allows us to make an animation of any sign without limitation, unlike another program called Sign Smith Studio 3.0. Sign Smith is another animation program that we use to make sign language sentences, but it has some limitations: it only has 800 words in its dictionary. Gesture Builder allows us to get around this limitation.
Some days we have experiment at the lab. Those days are fun because we can learn about another person’s background and ASL skills. A participant comes to the lab, and he or she signs answers to questions, describes videos, summarizes articles, and tells stories that we give to him/her.
When the person comes in, he/she has to fill a form and sign a few papers. After that, we start the experiment. There are four parts to the experiment: gloves setup, suit setup, eye-tracker helmet setup, and signing time. The gloves part is the longest part of the process. The signer has to pose their hands in specific shape that we tell him/her to do. We do one hand a time. It takes around 30 minutes a hand to set up the gloves. After the gloves, we put the sensors on the suit. When the suit setup is finished, we put the eye-tracker helmet on the participant, and we find the pupil and track it. After that it is show time: signing time. When the signing time is over, we take all the sensors and eyes trackers off. After that, we take off the gloves. When we are finished, then the person leaves.
Wesley's Blog Post
Wesley is a high school student who participated in a summer research experience at the lab during June-August 2009. He recorded a video blog entry about his experience, and he also used software at the lab to create an ASL animation expressing the same information as his video. He has also prepared an English version of his blog post which is included below.
English Version
Hi, my name is Wesley, and I am 18 years old. I will be senior this fall. I attend Lexington School for the Deaf. I am working at the LATLab, Linguistic and Assistive Technologies Laboratory at Queen College (City University of New York), for only 8 weeks in summer.
The purpose of the lab’s research is to analyze individuals’ use of ASL to improve ASL animations. We gather information about ASL by video camera from signers who come to the lab. After that, we use a program called Sign Stream to analyze the signers’ body, eyes, and eyebrow movement.
We also have a program called Sign Smith to make animations of sign language stories. However, Sign Smith has a limited ASL dictionary. That program’s signs don’t always match what you prefer it to say. Due to these limitations with the dictionary and matching the user’s preferences, we have another program called Gesture Builder 2.0. With that program, you can modify the signer in animations. You also can create your own custom signs that can be added to the Sign Smith program’s dictionary!
I have three favorite activities in the LATLab. I like to use the Sign Stream program to analyze each signer. Personally, I love analyzing things. So, I like to analyze the different type of ASL that signers use during our experiments. Most of the time, I determine how the signers move their head, eyes, and eyebrows at exact times. I also like to use Gesture Builder 2.0 because I love to play with the animation of signing. I like to make sign language animations become like real human’s signing. The best thing about LATLab is that we don’t work alone. We have partners who always help. There’s teamwork when making decisions, and this makes you feel comfortable.
There is a lot different equipment that we use during experiments. There is a lot of fascinating equipment that we use. I am most fascinated with calibrating the right and left gloves. Both of those gloves are put on the signer’s hands so that the computer can recognize the movements of the signer’s hand during the experiments. I am fascinated when both Professor Matt and Pengfei modify each of the joints on both hands. It takes them an hour to do both hands.
Video Tour of the Lab
Kelsey and Wesley are high school students who participated in a summer research experience at the lab during June-August 2009. They recorded a video blog entry in which they give a tour of the lab and describe some of its software. An English transcript of the video is included below.
English Transcript
Wesley: Umm. How can I make an animation on computer?
Wesley: But, How? What program? Who would do it? Who will? When? Where?
Wesley: oh! Hi! My name is Wesley Clarke. Here is Queen College “CUNY” City University of New York.
Wesley: This building is Computer Science
Wesley: Come on! I want to show you something. This is a secret.
Wesley: You know that this room is called LATLab. Let me show you the sign. Come here!
Wesley: Who are the researchers? Matt and Pengfei. Matt works very hard on this, and this is a 5 year planned project. Let me show you.
Wesley: Wow! That is a lot of work. Can you imagine that? Wow! Hold on, let me introduce my partner.
Kelsey: Hi! My name is Kelsey. (I am very sorry, but my partner acts silly sometimes and can really bug me.) Let’s go into the office. Come on!
Kelsey: We have 3 computers technically really 2 computers. Those two computers are connected. I allow Wesley to explain some of our equipment from here.
Wesley: Wow! See that? That is my partner. (She can be a pain in the neck but I like her.) She helps me a lot. See that? That is what I am talking about. This is not a one-person operation; we work together as a team.
Wesley: Let me show you the eye tracker. What is it for? It follows the eye ball when you are moving and signing. This is where you sit. Just to warn you: You have to always be careful when using the eye tracker (don’t touch it when signing) so we don’t have to keep re-programming it every time it is messed up. You can see your video here.
...
Wesley: How do we calibrate the eye tracker? Follow this and that. This equipment is for... Well, maybe Kelsey can explain…
...
Kelsey: That equipment tracks where the eye-tracker helmet is in space; this allows us to see where the head is when the person is signing.
...
Kelsey: One of the programs that I use is called Gesture Builder. This is my example. Wesley and I are defining animations for new verb signs for Gesture Builder. This is my example verb: “help.” That is one way to sign help. Another program we use, SignSmith, doesn’t allow you to move the character as much as Gesture Builder.
...
Wesley: That is very cool. Gesture Builder is used for signs that SignSmith doesn’t have. You can imagine any ASL verb signs, and then you can create a sign that matches your preference. This is a nice feature, but it also has a down side: it is hard work. You have to specify the movement of the animation to make the sign flow and move just right. The program I’m going to show you is cuter than hers because my program is called SignSmith. This program just signs words from its dictionary, but it has a limited set of signs to choose from. That is why we use Gesture Builder. Let me show you!
...
Wesley: My professor, Matt, told us to make an animation with a story about some topic we chose. It is very long but funny. Come on!
...
Wesley: See that? The signs are very limited that it can include. Right? Yes? Kelsey said yes. Gesture Builder has more capabilities while signing. I will show you how it works. Let’s see! See the small sign block. How does it work? You have to click each and every word. For example, “what” is one block, “do” is another block, and “you” is another block. You should understand that some of the words have default settings already for you. I mean, for example, like pointing forward or pointing right or left. That is my example to show you. Some of the face expressions are also automatic and the tokens... Actually, do you know what “token” means? A “token” means a point in space that you set up, like “my mom” (left side) and “my dad” (right side). That is an example of a “token.” Why do I set them up? When I continue to sign, I can point to either side and it will refer to whom I am talking about instead of my having to say “my mom” or “my dad” again. For example, “point right (Mom) love basketball and point left (Dad) love plumber.” Most deaf people find it easier to understand “tokens” than repeating “mom” and “dad.” That can be confusing.
...
Wesley: Now I will explain about this in the context of Gesture Builder… It is pretty much the same, but I should show it to you so that you understand it. This part of the program has two different controls: one is for the right hand and the other one for the left hand. For example, I can make it sign “hate” on the right hand. If you want two-handed version of the sign (“hate”), then this is how you do it. That is it!
...
Wesley: Now I will show you another room. We have three “rooms” here (areas of this room for each of us). We have our own “offices” (our own desk area). Oh! I forgot to introduce my other partner. Her name is Amanda; she is working on the SignStream program.
...
Wesley: See she is very focused on working. She is analyzing a person’s signs. See the video.
...
Wesley: Hi!
Amanda: What’s up?
Wesley: I am explaining what the LATLab is all about.
Amanda: I am Amanda Krieger from Gallaudet. I will be Senior this fall.
Wesley: Wow! She is our third partner. She is friendly too. Right now she is analyzing a person signing. She works very hard, and she is focusing on her work. Are you enjoying it, right?
Amanda: Yes, I was hired for this job.
Wesley: Yes, let see her work.
Amanda: Come on!
Amanda: See? The person sits and has everything on. We have to click and match the signs. Add lines for sentences, where the eyes are moving and where the person is looking. Also, there are many more details about this program. Come and look at all the lines.
...
Amanda: That is what we do here. Many people come and we take a video of them. All of us work to analyze the video afterwards, and we look for different signs in the performance. We’re looking for real ASL performances.
...
Kelsey: Thank you! I need a break from holding the camera. Come on and let me show you my video. I have the smallest “office.” The person who is signing this video is another person who works at the LATLab. His name is Jon. That is my work so far.
...
Wesley: What I’m doing is the same, but we are working on videos from different people. My work has the same motivation and purpose. We have to do add all of these lines to indicate linguistic information for all of the different stories. One thing that I love about this is that we can work on different stories. Opps! My bad: I am not finish yet with this video. Also, the whole process works faster when we all work on different stories… OK, I think we have covered everything.
Amanda: I guess you did cover everything.
Wesley: I wish that I could introduce my two professors, but they are not here. Oh well! Also, we have other equipment that we will show you on another day. Ok?
Video Tour of Equipment at the Lab
Kelsey and Wesley are high school students who participated in a summer research experience at the lab during June-August 2009. They recorded a video blog entry in which they describe the equipment at the lab. An English transcript of the video is included below.
English Transcript
Kelsey: Hello, my name is Kelsey Gallagher. Today, I will show you around the LATLab with all the equipment that we use. Now, I will show you my collaborator.
Wesley: Hello, my name is Wesley Clarke. I work with Kelsey here. Kelsey is a sweet girl, but sometimes I like to pick on her.
Wesley: She’s right about us giving you a tour of the LATLab’s equipment today. Remember the other day when you saw us on video; well, now we have all the equipment. (Taking a look at equipment.) This is all of the equipment that we use in LATLab.
Kelsey: The first thing that we will show you is the spandex shirt.
Wesley: This is the spandex shirt. Wow, it is very tight but it is all right. Do I have to zip the shirt up?
Kelsey: No, you don’t need to zip the shirt up.
Wesley: The purpose of this shirt is to allow us to stick a lot different sensors on to a person wearing it.
Wesley: There are a lot of sensors. (Pointing at sensors.) What you see now is just a few of them, but when we have experiment, we use a lot of sensors.
Wesley: We’ll place these sensors all over the spandex shirt. That’s a lot of work. These sensors allow the computer to recognize your body’s movements. Suppose that none of sensors were placed on the spandex shirt, then how would the computer recognize your body movement?
Wesley: Now the second step (for setting up the equipment) is to put on the gloves.
Wesley: This kind of looks like Michael Jackson’s glove. (Pointing at white glove.)
Wesley: Now the third step is to put on the sensor gloves. This is what I am talking about. This glove will recognize your hand movements during the experiment.
Wesley: Let me show you. You have to be very careful. You can’t touch the brown surface of these gloves. But you can touch the black surface. You cannot pull this glove on quickly like a normal glove. No way. This is much different than normal gloves. Let me show you.
Wesley: (Pointing at glove.) Let me show you how to put this on.
Wesley: That’s it. It is cool and very tight. It also has to be connected to the computer that I just mentioned.
Wesley: During the experiment, when we’re setting up the gloves, you have to follow the hand shapes that are displayed on the computer screen. For example, if the computer displays shows a “B” handshape, then you have to follow that hand shape with your hand. This helps us set up the gloves. Before we adjust the glove settings, your hand might be in a nice and perfect hand shape; however, the computer at first isn’t able to follow the exact shape of your hand.
Wesley: Two other researchers here, Matt and Pengfei, have to fix each joint on the glove in the computer so that it matches your hand shape precisely. It takes them about an hour to complete both gloves. I’m only going to show you one hand for now. Let me take this off.
Wesley: I need your help.
Wesley: Wow, did you see that? How hard it is to take the glove off? You had to be very careful when removing these gloves. Let me take this off.
Wesley: Now, what we will do next? Put on the neck strap around my neck. This purpose of this is so that we can stick sensors on the neck. This is just like how we stick sensors on the spandex shirt. Except now, the neck sensor allows the computer to recognize your neck movement.
Wesley: The next thing is eye-tracker helmet. Let me show you. This is eye tracker helmet.
Wesley: The eye tracker helmet has a little mirror near your left eye. The purpose of this is to follow your eye movements during experiment.
Wesley: It also has video camera on the middle of your forehead.
Wesley: That is all equipment that we use during experiment.
Wesley: I guess that is it.
Wesley: At the end of the experiment, the person would take the spandex shirt off. That is it.
Wesley: Wait a minute, let me introduce my professor.
Wesley: We might scare him and the other researchers when we enter the room because they have all been working very hard.
Wesley: I will introduce Pengfei first... No, I will introduce Matt first.
Kelsey: Yes
Wesley: Ok, come on. Be very quiet. (Sneaking into the room.) (Didn’t actually surprise them.)
Matt: Hi, my name is Matt Huenerfauth. I am professor here at Queen College.
Wesley: Now I will introduce Pengfei. She is a student, right?
Matt: Yes, she is a PhD student. You’re right.
Wesley: She is a PhD student.
Pengfei: Hi
Matt: This is Pengfei Lu. She is student at CUNY.
Wesley: That’s it.