Head-mounted Myoelectric Mouse

Technological realization: With technologies of motion sensing, myoelectric sensing and speech recognition, etc., the equipment can be used to:
1. Convert the head motion to mouse motion with gyroscope to free both hands to help those who have difficulty in moving and have lost single/double arm(s).
2. Realize mouse single-click and double-click of mouse by detecting the holding-on of tooth masticatory muscle with the muscle electric sensor.
3. Select the control/input mode for the voice. The control mode can realize command control, e.g., “copy” and “paste”, etc.; the input mode can convert the voice to text, to realize fast control and input.
4. Realize functions of sitting posture detection and cervical spondylosis prevention, etc. with motion sensor;

Parts in need

• Arduino Leonardo Microcontroller
• Gravity : Digital Buzzer for Arduino  
• Gravity: Analog EMG Sensor by OYMotion
• 6 DOF IMU Shield For Arduino
• Speech Module

Step I: Test and install the myoelectric sensor

I was excited when I got the OYMotion myoelectric board. OYMotion includes a dry electrode to realize the medical grade accuracy.
This is the original module.

To reduce the size, we will remove the headset wire part and transfer with fly wire.

How to connect the wire?

It is very easy: VCC to 3.3V, GND to GND, data line A0 to AD . The data can be read only with a word: data = analogRead(A0);
However, how to realize single click and double-click of tooth? Firstly, ArduinoLeonardo developed by HID shall be familiarized approximately. After the data is acquired with Arduino, the single click and double-click of mouse can be realized through waveform analysis! See the codebase part for the algorithm of waveform analysis. Let’s see two oscillograms at first. Both oscillograms represent the process of one-time teeth gnashing + long-time gnashing. The first one is the original data, and the second is the data processed after being added with algorithm (red line represents the process of pressing the mouse down).

Step II: Install the gyroscope

       It’s easy to sense the head motion with gyroscope. I selected the existing gyroscope module JY901 on the market to directly output the angle data of gyroscope. The mouse movement can be calculated through angle. See the code for details of algorithm. Note: According to the experience, the module must be at the horizontal/ vertical position and head top part is the best!

So what do we use to make the support? The support shall be flexible, can clamp the myoelectric sensor and shall be suitable for wearing! Then, I can only sacrifice my headset.

Step III: Voice module

The voice module may sound difficult. Actually, it is not complicated. You will understand when you see the Arduino operation provided on the official website. Actually, the recognition is made through pinyin.
The voice module is connected with Arduino through SPI.  Please see http://www.waveshare.net/study/article-11-1.html for details.

Step IV: Connection of other small parts:

Small parts such as buzzer and LED are also needed for prompting message. One LED wire is connected to GND and another LED wire is connected to IO 13 port. Except for VCC GND, the data line of buzzer is connected to IO7.
The specific overall installation method: 1 The myoelectric module is in the cochlea of headset and shall fit with the face as far as possible when being worn; 2. IMU module is placed at the head top to sense the head motion; 3. The main control module Leonardo is placed at the ear side; 4. The voice module is placed at the front of main control module and can be closer to the speaker; 5. The buzzer is hided in the main control plate. LED lamp is placed at the front of voice module. After wearing, you may also see it with eyes.

Complete Image

The design sketch is as follows:

#include #include #include #include #include #define PRINT_RAW_DATA 0 #define PRINT_STD_DATA 0 bool mouse_disabled=false; void setup() {  Serial.begin(9600);                        //Equip 9600                voice_init();  mouse_move_init();  musle_click_init();  Mouse.begin();  Keyboard.begin();  Serial.print("Initialized\n"); } void loop() {  handle_voice();  if(!mouse_disabled){    mouse_move();    handle_musle_click();  } } /*---------------------------------------------------------------mouse movement-------------------------------------------------------------------------*/ float Angles[3]; float xLast,yLast; bool right_clicked=false; bool left_click=false; float view_center; float view_angle=30; int shake_x,shake_y; #define SHAKE_LIMIT 16 void mouse_move_init(){    Serial1.begin(9600); } void mouse_move(){  int x,y;  float xDelta,yDelta;  float xNow,yNow;  float X_RATE=0.06;//0.05;  float Y_RATE=0.04;//0.05;  get_IMU_data();  xNow=Angles[2];  yNow=Angles[0];    if(xLast==0){    xLast=xNow;  }  if(yLast==0){    yLast=yNow;  }  xDelta=xNow-xLast;  yDelta=yNow-yLast;  x=xDelta/X_RATE;  y=yDelta/Y_RATE;  if((x!=0 || y!=0)){    if(xLast*xNow<0){      x=xLast/abs(xLast)*(360/X_RATE-abs(x));    }    int out_x=x, out_y=y;    if(left_click==true){         //do not let mouse move when left button pressed, in case it's during right pressing.      shake_x+=x;      shake_y+=y;    }else{      while(abs(out_x)>120){        out_x=out_x/abs(out_x)*(abs(out_x)-120);        Mouse.move(-(out_x/abs(out_x)*120), 0);      }      while(abs(out_y)>120){        out_y=out_y/abs(out_y)*(abs(out_y)-120);        Mouse.move(0,-(out_y/abs(out_y)*120));      }      Mouse.move(-out_x, -out_y);      Serial.print("xNow=");Serial.print(xNow);Serial.print(" yNow=");Serial.print(yNow);      Serial.print(" xDelta=");Serial.print(xDelta);Serial.print(" yDelta=");Serial.print(yDelta);Serial.print(" x=");Serial.print(x);Serial.print(" y=");Serial.println(y);    }  }    xLast=xLast+((int)(xDelta/X_RATE))*X_RATE;  yLast=yLast+((int)(yDelta/Y_RATE))*Y_RATE; } void get_IMU_data(){  if(Serial1.available()) {    JY901.CopeSerialData(Serial1.read()); //Call JY901 data cope function    for(int i=0; i<3; i++){      Angles[i]=(float)JY901.stcAngle.Angle[i]/32768*180;      //Angles[i]=(float)JY901.stcGyro.w[i]/32768*2000;    }  } } /*-------------------------------------------------------voice detect--------------------------------------------------------------------*/ #define buzzerPin 7 #define VOICE_ARRAY   16 bool start_switch = false; int current_mode=0; VoiceRecognition Voice;                         //classify a voice recognition object char *voices[VOICE_ARRAY]= {"mushroom cloud",            //0 "input mode",          //1 "command mode",       //2 "new open",               //3 ctrl+n "close",                //4 alt+f4 "copy",                 //5 ctr+c "paste",               //6 ctr+v "good weather",        //7 "maker contest",//8 "tou kong shu ru she bei", //9 "qiao",                 //10 "I am so happy",            //11 "everything is fine",           //12 "turn on the mouse",        //13 "disable the mouse",       //14 "normal mode"      //15 }; void voice_init(){  pinMode(buzzerPin, OUTPUT);  Voice.init();                               //Initial VoiceRecognition mode    for(int i=0; i2){      Keyboard.print(voices[voice_id]);    }  }  if(current_mode==2){    //command mode      char ctrlKey = KEY_LEFT_CTRL;      char altKey = KEY_LEFT_ALT;            switch(voice_id){        case 3:          Keyboard.press(ctrlKey);          Keyboard.press('n');          delay(10);          Keyboard.release('n');          Keyboard.release(ctrlKey);          break;        case 4:          Keyboard.press(altKey);          Keyboard.press(KEY_F4);          delay(10);          Keyboard.release(KEY_F4);          Keyboard.release(altKey);          break;        case 5:          Keyboard.press(ctrlKey);          Keyboard.press('c');          delay(10);          Keyboard.release('c');          Keyboard.release(ctrlKey);          break;        case 6:          Keyboard.press(ctrlKey);          Keyboard.press('v');          delay(10);          Keyboard.release('v');          Keyboard.release(ctrlKey);          break;      }  } } int voice_control(){  int id = Voice.read();  if(id >= VOICE_ARRAY || id < 0){    return -1;  }  Serial.println(voices[id]);  switch(id)                            {    case 0:                                     //start the order        start_switch = true;        Serial.print("Start command\n");        tone(buzzerPin, 5000, 200);        delay(350);        tone(buzzerPin, 5000, 200);        break;    case 1:                                     //input mode        if(start_switch){          current_mode=1;          start_switch = false;          Serial.print("input mode\n");          tone(buzzerPin, 5000, 200);        }        break;      case 2:                                     //command mode        if(start_switch){          current_mode=2;          start_switch = false;          Serial.print("Command mode\n");          tone(buzzerPin, 5000, 200);        }        break;    case 15:                                     //normal mode        if(start_switch){          current_mode=0;          start_switch = false;          Serial.print("normal mode\n");          tone(buzzerPin, 5000, 200);        }        break;    case 14:        if(start_switch){                         //disable the mouse          mouse_disabled=true;          start_switch = false;          Serial.print("Mouse disabled\n");          tone(buzzerPin, 5000, 200);        }        break;    case 13:        if(start_switch){                         //turn on the mouse          mouse_disabled=false;          start_switch = false;          Serial.print("Mouse turn on\n");          tone(buzzerPin, 5000, 200);        }        break;    default:        //Serial.print("nothing...");        start_switch = 0;        break;  }  return id; } /*----------------------------------------------------------- musle click---------------------------------------------------------------------------*/ int sensorPin = A0;    // select the input pin for the potentiometer int sensorData = 0;  // variable to store the value coming from the sensor int errorPin = 13; bool errorHappen = false; int calib_count=0; long calib_data=0; unsigned long errorTime=0; #define DATA_SIZE 15 int musle_data[DATA_SIZE]; int diff_data[DATA_SIZE]; #define MIN_VALUE 150 #define MAX_VALUE 600 #define STD_ABNORMAL 250 #define STD_VALUE 10 #define STD_EDGE 3 unsigned long press_timestamp=0; unsigned long release_timestamp=0; int show_press_real=0; #define PRESS_DELAY 40 #define RELEASE_DELAY 15 #define RIGHT_BT_DETECT 600 #define SAMPLE_DELAY 10 unsigned long last_sample_time=millis(); void musle_click_init() {  Serial.begin(9600);  pinMode(errorPin, OUTPUT);  digitalWrite(errorPin, HIGH);  errorTime = millis();  Mouse.begin(); } void handle_musle_click() {  if(millis()-last_sample_time > SAMPLE_DELAY){    sensorData = analogRead(sensorPin); #if PRINT_RAW_DATA    Serial.println(sensorData); #endif    for(int i=0; iSTD_VALUE){    show_press=30;  } #if  PRINT_STD_DATA  Serial.print(show_press);Serial.print(" ");  Serial.print(show_press_real);Serial.print(" ");  Serial.println(std_value); #endif  if(std_value>=STD_ABNORMAL || std_value<0){    errorTime = millis();    digitalWrite(errorPin, HIGH);    errorHappen = true;    calib_data=0;    calib_count=0;  }  if(errorHappen){    if(std_value=0){   //when error happened, make sure musle std data is below STD_VALUE for 1s.      calib_data+=std_value;      calib_count++;    }else{      errorTime = millis();    }    if(millis()-errorTime > 1500){      digitalWrite(errorPin, LOW);      errorHappen = false;      calib_data=calib_data/calib_count;  //re-calibrate the musle average data.    }  }  if(errorHappen){    release_timestamp=millis();    press_timestamp=0;    show_press_real=0;    if(Mouse.isPressed(MOUSE_LEFT)){      Mouse.release(MOUSE_LEFT);    }  }  if(!errorHappen){        if(std_value>calib_data+STD_EDGE){      if(press_timestamp==0){        press_timestamp=millis();          }else{        unsigned long current_time = millis();        if (current_time-press_timestamp>PRESS_DELAY && current_time-press_timestamp<=RIGHT_BT_DETECT) {       //make sure left-click can be detected.          left_click=true;        }        if(sqrt(shake_x*shake_x+shake_y*shake_y)>SHAKE_LIMIT){  //when mouse moved, let left button "press"          if(!Mouse.isPressed(MOUSE_LEFT)){            Mouse.press(MOUSE_LEFT);          }          left_click=false;        }else{          if(current_time-press_timestamp>RIGHT_BT_DETECT && right_clicked==false){ //mouse did not move, and is pressed bigger than threshold value.            Mouse.click(MOUSE_RIGHT);            right_clicked=true;            left_click=false;          }        }          release_timestamp=0;          show_press_real=calib_data+STD_EDGE;      }    }else{      if(release_timestamp==0){        release_timestamp=millis();      }else if (millis()-release_timestamp>RELEASE_DELAY) {       //when musle released, clear all the flags.          if(left_click==true){            Mouse.click(MOUSE_LEFT);          }          if(Mouse.isPressed(MOUSE_LEFT)){            Mouse.release(MOUSE_LEFT);          }          right_clicked=false;          left_click = false;          shake_x=0;          shake_y=0;          press_timestamp=0;          show_press_real=calib_data;      }    }  } } long int get_std(){  int average = 0;  int sum = 0;  long int std = 0;  for(int i=0; i MAX_VALUE){      return STD_ABNORMAL;    }  }  average = sum/DATA_SIZE;  for(int i=0; i