This week I focused on setting up the Myo and getting it to run in Unity.

The package that comes with the Myo development kit makes this easy as it comes with an example scene element (a ball on a stick – which is literally just that.) And as bunch of code that allows the Myo to rotate this ball on a stick. It also lets you change the texture of it and cause vibration with some gestures.

My goal this week was to get the scene working with sound and get an android build working on my phone.

Android compatibility

The first test was simply using the scene and compiling it for android. After doing this and launching it on my phone, the first issue emerged that, of course, my phone did not detect the Myo, even with Bluetooth switched on. I tried another method of using a USB extension cable (micro USB into the charger port and a normal USB out) and connect the Bluetooth dongle to that. I did this as some research revealed that the Myo needs the particular dongle hardware to function at all.

In desperation,  I contacted my trusty Myo contact via email, but got a vacation notice as a reply. I then decided to go on the Myo developer forums and ask the question of how I would go about getting it to work on my android. A Myo developer replied saying that the Myo is supported on android,  but not Unity. However, it would be supported in the future.

This was unfortunate because I had not allowed time in my schedule to get the game to work on android using Java or C++, so I would have to simply make a prototype on Windows using Unity and present it with the intent on going mobile once Unity is supported with the Myo on android. Not a complete loss, but it’s still a bit of a damper on my hopes and dreams of a fun audio game. It also means I have to re-do some of my test plans as they relied on a questionnaire about a mobile specifically.

Implementing audio

The next goal was making the game play audio. This was simple as Unity has an inbuilt audio system. So it was as simple as importing a wav file to play. I used a fireball dart sound as the sword swing. To get it to play, I wanted it to somehow detect that I am swinging the sword and then play the swing sound. To do this, I first tried detecting the difference between 2 quaternions from the previous frame and the current frame. This turned out to be more complicated as the Myo seemed to not always send data to unity at a constant rate, so the resulting difference seemed to be a random number or 0. This, I assume, is because the Myo does not send data at the same rate as the game update ticks, and so comparing the 2 differences would try and compare with the same position withing 1 Myo update a the difference would be 0.

A way to fix this would be to have a function that runs “onMyoUpdate()” and do the calculations there. However this may make the game go out of sync and make calculations with the game world harder. Also when working with audio, it is a lot easier to hear tiny differences in the audio volume or pitch than to visually see something, and so having some updates being skipped may cause the audio to be “choppy”.

In the end, I simply coded around it by first getting the difference of the vector of the end of the sword (the sword doesn’t actually move, but rather changes angle only, so the sword end would accurately represent a velocity of it spinning) of the last frame and current frame, and then added that onto a “power” variable. This would also decrease each frame slightly but never go past 0. This means that even if the difference is 0, there is still a possibility that the power value is higher than 0.

Using this power variable and a cool-down of about half a second,  I got the sound to play when the power was higher than a certain level, but also change the volume of the sound by the amount of power. This effectively made the long swing sound fade out quickly if the user stops swinging suddenly. The cool-down was added so the sound wouldn’t start to play continuously as the user swings. A better option than a cool-down would be a variable that calculates if the sword is travelling in a certain direction each frame, and if it changes, would cause it to reset some flag, telling the script that the sound can be played again. However, at this point in time we can go with the cool-down option and see how the testers go with the game.

Below is a video of the result, also showing the mentioned 0s in the console coming from the Myo.

What’s next?

The next thing I will be implementing will be a sound for when the sword collides with the center of the scene which would represent an enemy.

After this, I will implement a scoring system, where the enemy will have 5 hit points, and be blocking for some of the time. The blocking will be signified by a “shield being hit” sound with echo. The enemy will also make a sound for when he is about to swing. When this happens, you will be expected to block. To block, the player must hold his or her arm vertically and make a fist gesture. When this is done correctly, you will hear the same sort of sound as the enemy (but needs to be different, to avoid the player thinking the enemy blocked instead) this would be the core of the game.

Ideas for future

As the game cannot be ported to mobile using unity, we will not be seeing a mobile app in this version. However, once mobile Myo is supported in unity, the game could expand into a jogging adventure app where an adventurer can go on quests to slay enemies and get loot. The catch is that they have to jog at least 2 KM (using GPS or a treadmill) and fight the enemy before returning home.

Research update

As described in a previous post, I’ll need to research other motion capture and gesture capture controls like the Razer Hydra, Perception Neuron and the Nintendo Wii controller.

I will then look into other mobile games and how they do audio. Games would be “Papa Sangre” and “Zombies! Run!“. Some notable features of Papa Sangre is that they muffle audio from things that are behind the player. This is not a feature in unity, although it can be implemented. One issue is that the Myo game I am proposing has the issue of assuming the player’s head is facing towards the enemy as there is no head tracking. This will need to be experimented with.

Test cases

• Audio panning and muffling helps or causes weird experience ?
• Cool-down on sword. Is it annoying if you swing at a weird time? Should it play based on velocity or based on change of direction?
• How fun is the game in general?