Saturday, June 2, 2012

Audio mix and record in Android

iOS offers, among its frameworks, many interesting features that allow to create audio tracks by simply mixing multiple tracks together. You can use Audio Unit and its methods, as described here:

http://developer.apple.com/library/ios/#documentation/AudioUnit/Reference/AUComponentServicesReference/Reference/reference.html#//apple_ref/doc/uid/TP40007291

But what if you need a similar result on Android? Android does't offer such feature in its audio framework. So I've spent a couple of days on google groups and stackoverflow, reading unanswered questions of android devs searching for a similar functionality on the Google mobile platform, or developed and released by third party contributors and external devs.
It appears there isn't nothing available.
So I've studied the problem and the tools I had to solve it. First let's see what possibilities the platform offers to play files.
Android audio framework consists of these main classes for audio playback:


  • MediaPlayer: useful to play compressed sources (m4a, mp3...) and uncompressed but formatted ones (wav). Can't play multiple sounds at the same time. [HIGH LEVEL methods]
  • SoundPool: can be used to play many raw sounds at the same time.
  • AudioTrack: can be used as SoundPool (raw sounds), but need to use threads to play many sounds at the same time. [LOW LEVEL methods]
I've found that AudioTrack works fine to play uncompressed raw data, and if you want to play multiple sounds at the same time, you can create different threads and start the playback in an asynchronous fashion.
Unluckily this is not always precise: sometimes you can experience a delay before a certain sound is played, and in such cases the final result is far from acceptable.

Another option is to mix sounds before playing them. This option offers you a nice plus: you obtain the mixed sound that is ready to be stored on file. If you mix sounds with SoundPool for instance, then when you play it, you cannot grab the output and redirect it to a file descriptor instead of to the audio hardware (headphones or speaker).
As mentioned at the beginning, there is no ready solution for such problem. But actually we will see the solution is rather trivial.

Before delving in the details of how 2 sounds can be mixed together, let's see how can we record a sound on Android. The main classes are:


  • MediaRecorder: sister-class of MediaPlayer, can be used to record audio using different codecs (amr, aac). [HIGH LEVEL methods]
  • AudioRecord: sister class of AudioTrack. It records audio in PCM (Pulse Code Modulation) format. It is the uncompressed digital audio format used in CD Audio, and it is very similar to .wav file format (the .wav file has 44 bytes header before the payload). [LOW LEVEL methods].


AudioRecord offers all the features we want be able to control: we can specify the frequency, the number of channels (mono or stereo), the number of bit per sample (8 or 16).


In the fragment of code posted above, there is a simple function that can be used to record a 44.1khz mono 16 bit PCM file on the external storage. The function is blocking so it must be run on a secondary thread; it continues to record until the boolean isRecording is set to false (for example when a timeout expires or when a user taps on a button).

And now comes the most interesting part: how to mix two sounds together?

Two digital sounds can be mixed easily if files have the same features (same number of channels, same bit per samples, same frequency). This is the simplest scenario and is the only one I'm covering in this post.
Every sample in such case is a 16 bit number. In java a short can be used to represent 16 bit numbers, and infact AudioRecord and AudioTrack work with array of shorts, which simply constitute the samples of our sound.

This is the main function used to mix 3 sounds together:


There are some complementary methods I'm not posting here because this post is already too long :) but these are some small hints of what they do:

  • createMusicArray reads the stream and returns a list of short objects (the samples)
  • completeStreams normalizes the streams by adding  a series of '0' shorts at the end of smaller files. At the end the 3 files have all the same length.
  • buildShortArray converts the list in an array of short numbers
  • saveToFile saves to file :)
The key point in the method is that we sum every sample together. We normalize the short to a float [-1,1] so we dont have under/overflow issues. At the end we reduce a bit the volume and we save it in the new array. That's it!

Of course this is the simplest scenario; if the samples have different frequency we should do other computations. But I think most of the time we want to mix sounds we can also control how they are recorded thus reducing its complexity a lot.

Once we have a PCM mixed sound, it can be transformed in a .wav file so that every player can read it. EDIT: as many people have asked me some more help, below it is the code snippet to build a short array from a file containing a raw stream.

40 comments:

  1. Hi Adolfo,

    Your post is really interesting. I've spent a few days trying to mix voice and music. Could you post the complete code to test it?

    Thanks in advance.

    ReplyDelete
  2. Hi,

    unfortunately I cannot post the complete code because it is copyrighted. I can just post snippets to give an idea of the main concepts.
    Basically the missing methods are trivial, I can help you if you provide some details on the issues you're facing.
    Regards!

    ReplyDelete
  3. Hi adolfo,

    can you please send the whole code to my email. I really need this. I will be very thankful to you.

    Regards!
    maverick.sahil8@gmail.com

    ReplyDelete
  4. Please send me the whole code of MixFiles that will very useful tome or any other link like as..
    Thank you.

    Regards
    yesha

    ReplyDelete
  5. please provide the code for buildShortArray.. it will be really helpful
    Thank you

    ReplyDelete
  6. As i have received many comments, I have decided to provide some more code, specifically the buildShortArray, that might be a bit tricky.

    Cheers

    ReplyDelete
    Replies
    1. hi, adolfo,
      please provide the full code of buildShortArray ...
      thanks..

      Delete
  7. Hi adolfo,

    can you please send the whole code to my email. I really need this. I will be very thankful to you.

    Regards!
    tungpetit@gmail.com

    ReplyDelete
  8. This comment has been removed by the author.

    ReplyDelete
  9. This comment has been removed by the author.

    ReplyDelete
  10. This comment has been removed by the author.

    ReplyDelete
  11. Hi adolfo,

    Could you send the whole code to my email.
    Please....

    I really need this.

    Regards!
    soccerjung@naver.com

    ReplyDelete
  12. Hi was not able to play the audio file created using AudioRecord.

    ReplyDelete
  13. Hi adolfo,
    I have implemented the code. Now i can record the sound and can mix them without any error. The problem is when i mix two audio of 5 secs, i got the output of 2 secs audio only, and also it is very noise. I cant hear anything.
    Please help me.

    ReplyDelete
  14. hi Jithin, if you are able to merge two sounds then you shouldn't hear noise. Concerning the length of sound: usually there is a delay, that can be also of one second or even more, between the press on the record button and the moment the system starts recording. This depends on the implementation of drivers, on hardware...
    Concerning the noise: again if you are able to mix sounds, and you can listen to the mixed output, then it shouldnt change based on the length of the recording. If you are not able to listen to the mixed sound (whatever length it is), then you might probably want to swap the samples recoded. I had some problems too because i forgot to swap the samples. Samples are recorded as 16bit, so each sample is a short (16bit). You need to transform it in a couple of bytes, then swap them. It is a problem of how sounds are sampled (big endian vs little endian). You can do some experiments with the sounds and the sample file you have recorded through a useful program called audacity. http://audacity.sourceforge.net/ there is an option to play sound as little endian or big endian. So you can start from there. :)

    ReplyDelete
    Replies
    1. This has always bugged me as some peoples answer is "you have to mix them yourself" via arrays and generate the resulting sound on your own --when the sound is coming out somehow mixed and generated from the darn thing's speaker already! Why would you have to re-invent what's already happening at some level on your android?

      Delete
    2. It is not happening actually. What I want is to record the mixed sound on a file. I don't want to just stream the mixed sound but be able to "capture it" in a file. If you just want to stream a mixed sound you can use the SoundPool class as stated at the begin of my post.

      Delete
  15. Hi Adolfo
    I don't understand what does "32768.0f" mean?
    how does it correlate to 3 samples sound?
    what about 7 samples sound?
    thanks.

    ReplyDelete
    Replies
    1. If you have 7 sounds it would be the same. It is not correlated to the 3 sample sounds. Basically, in that line of code I am normalizing the number (which is a java short primitive). A short in java is a number between -32767 and 32767. So by dividing it by 32768 I will get a number between -1 and 1 :)

      Delete
  16. Hi Adolfo
    I do not speak english very well. But, I have a question. So I ask you. I have one long .wav file and short .wav files like sound effects. I want sound effects let in long file at different time. It was possible? If it was possible, then How? thanks ㅜㅜ

    ReplyDelete
  17. hi Adolfo. This post really helps. I'm developing an App that can record your voice while playing 2 mp3 files for instrumental and vocal guide. this App is like Karaoke app. My main problem is I can't play the recorded file and instrumental file synchronously after saving it. Hope you can Give me code that can Mix the instrumental file and the recorded file during recording. I badly needed it. Please help. Thank you. Regards -Ariel

    ReplyDelete
    Replies
    1. Hey Ariel, how are you, I have a same case, did you get it solved this problem?, please let me know if you would., Thanks

      Delete
  18. Try with this for mixing (i dont know how to save short array to wav, then i play it):


    private void mixFiles(){
    try {

    InputStream is1 = new FileInputStream(new File(Environment.getExternalStorageDirectory(), "/Android/data/blablabla.mp3"));
    InputStream is2 = new FileInputStream(new File(Environment.getExternalStorageDirectory(), "/Android/data/blablabla2.mp3"));
    InputStream is3 = new FileInputStream(new File(Environment.getExternalStorageDirectory(), "/Android/data/blablabla3.mp3"));

    text.setText("haciendo shorts");

    short[] music1Array = bytetoshort(convertStreamToByteArray(is1));
    short[] music2Array = bytetoshort(convertStreamToByteArray(is2));
    short[] music3Array = bytetoshort(convertStreamToByteArray(is3));

    short[] output = new short[music1Array.length];
    for(int i=0; i < output.length; i++){

    float samplef1 = music1Array[i] / 32768.0f;
    float samplef2 = music2Array[i] / 32768.0f;
    float samplef3 = music3Array[i] / 32768.0f;

    float mixed = samplef1 + samplef2 + samplef3;
    // reduce the volume a bit:
    mixed *= 0.8;
    // hard clipping
    if (mixed > 1.0f) mixed = 1.0f;
    if (mixed < -1.0f) mixed = -1.0f;
    short outputSample = (short)(mixed * 32768.0f);

    text.setText("reproduciendo1");
    output[i] = outputSample;
    }
    play(output);
    escribir(output);
    } catch (NotFoundException e) {
    toast("notf");
    e.printStackTrace();
    } catch (IOException e) {
    toast("ioex");
    // TODO Auto-generated catch block
    e.printStackTrace();
    }
    }

    //==============================================================

    public short[] bytetoshort(byte[]bite){
    // Grab size of the byte array, create an array of shorts of the same size
    int size = bite.length;
    short[] shortArray = new short[size];

    for (int index = 0; index < size; index++)
    shortArray[index] = (short) bite[index];

    return shortArray;

    }

    //=======================================================

    public byte[] convertStreamToByteArray(InputStream is) throws IOException {

    ByteArrayOutputStream baos = new ByteArrayOutputStream();
    byte[] buff = new byte[10240];
    int i = Integer.MAX_VALUE;
    while ((i = is.read(buff, 0, buff.length)) > 0) {
    baos.write(buff, 0, i);
    }

    return baos.toByteArray(); // be sure to close InputStream in calling function

    }

    //=============================================================

    public void play(short[] music){
    AudioTrack at=new AudioTrack(AudioManager.STREAM_MUSIC, 48000, AudioFormat.CHANNEL_OUT_MONO,
    AudioFormat.ENCODING_PCM_16BIT, 48000*7 /* 1 second buffer */,
    AudioTrack.MODE_STREAM);
    at.write(music, 0, music.length);
    at.play();
    }

    ReplyDelete
  19. Hi ,

    I want to make like beats reorder. is it possible using this post? Mixing of my voice and background music. is possible using this code? like Mp3 player with voice recorder

    ReplyDelete
  20. Hi ,

    I want to make like beats reorder. is it possible using this post? Mixing of my voice and background music. is possible using this code? like Mp3 player with voice recorder

    ReplyDelete
  21. Hi, please help!
    Do you have this perfect code?
    And audio files' type is what?

    ReplyDelete
  22. Hi,please help !
    Which file extension can be used as input to the mixFiles() method.
    Can it be done on byte basis rather than short?

    ReplyDelete
  23. hi please tell me how to concatinate the two audio file ???

    ReplyDelete
  24. Nice post. I was in need of Android app, which allows recording calls, and total recall is the user friendly app which allows to record a phone call with clear recording and you can get this app in just 50% less price along with 90 days free demo pack.

    ReplyDelete
  25. Excellent post, thanks for collating the information. I have been searching google and yahoo for information related to this and it led me to your blog! :-)
    ipad app development company| web development company

    ReplyDelete
  26. Please send me a full code because i am developing a music app.in which record a voice with background playing music.
    so please help me and send me to a full code in my email id:-abhishek.pixlrit@gmail.com

    ReplyDelete
  27. Thanks for sharing this valuable information with us.Each and every point you have mentioned is really very important.
    custom web development Melbourne | web design and development company

    ReplyDelete
  28. send me to a full code in my email id:-s.sureshkumar2@gmail.com

    ReplyDelete
  29. Hi,
    Could you please send me the sample code . just record voice +play music+mix this two. Just simple code. My email id is mmaid786@gmail.com
    Kind Regards,
    Md Maidul Islam

    ReplyDelete
  30. Hi Adolfo,

    I think there is a bug in this code.
    On line 49 of sampleToShortArray(), the parameters of the loop indicate that it should keep running while "i" is less than the length of the array.
    However, on lines 58 and 61, you are referencing the "i+1" element of the array.
    Won't this cause an OutOfBounds Exception when "i" references the last element of the array?

    ReplyDelete
  31. Hi,
    Can you give me a demo code how to mix more then two audio files and save it .mp3 file, please it's very important for me.

    my Email id is : prabhanshu946@gmail.com

    ReplyDelete
  32. voice recorder is a convenient and simple tool that can be used for audio recorder and voice recording. Their are many new features comes in voice recorder which help you in recording the voice.Heinrich limited provide high qualitative products of safety and technology.

    ReplyDelete
  33. Check this call recorder App it also fail to work in many devices i am trying to make it work for motorola devices but its not supporting

    ReplyDelete
  34. Can you please send the whole code to my email. I will be very thankful to you. Please...

    Regards!
    mahmutadana@hotmail.com

    ReplyDelete