Sign in to follow this  
Followers 0
MikeatW3S

To synchronize vertical scrolling with audio file

26 posts in this topic

I have a website with text, and I have an audio file of a reading of that text (adding my own emphasis in the author's voice). What I'd like to do is synchronize the vertical scrolling of the text with the audio file so that the text that is being read out loud also appears in the middle of the window. And if anyone should drag the vertical scroll-bar to some text, then the audio timing will adjust to that portion of the text be read out loud. I'm not really sure how to go about that - what ways of do that are there?

 

I have seen techniques to simply scroll vertically at whatever speed I would desire. And I suppose I could just match the timing to coincide with the audio. Or I have seen where I can assign each paragraph with its own id="xxx". Perhaps I could periodically get the currentTime() of the audio and force the paragraph by id to the middle of the window (no matter what the size of the text or window). What DOM references and javascript statements would I have to look up to do that? Or if anyone should have a better idea, I would certainly like to hear about it.

 

Thanks.

Share this post


Link to post
Share on other sites

This sounds complicated.

 

To begin with, a basic solution could be to change the scrollTop of the box containing the text based on ratio of how much was played and how much is left to play. There's a problem with this: You can't be certain that the words in the audio are evenly distributed throughout the file. Perhaps at some point he's talking faster than at other points.

 

I think many audio formats allow embedded subtitles, I haven't investigated much in that field but it's something I would do if this were the task assigned to me.

 

Something I certainly would have to investigate is the <audio> element's API to figure out how much is played and how long the file is. A search for "MDN Audio API" will probably find something. I like the MDN (Mozilla Developer Network) because they have very good documentation for Javascript features.

 

What you might need to is load data from a file that associates particular times in the audio file with particular written sentences. Each time one of these time intervals is reached the scrollTop property of the box containing the text is updated with the new position. I know that scrollTop can be changed in most browser, but it probably is better to use the .scrollTo() method instead.

1 person likes this

Share this post


Link to post
Share on other sites

Thank you, Foxy Mod. You've been a great help in this project.

 

One problem I will have is that I cannot anticipate the size of the text and the size of the window in which it resides. The user may change all that. What I'd like to be able to do is to identify before any scrolling starts what the vertical coordinate of the starting text would be and what the vertical coordinate of the ending text would be. I know I can identify paragraphs with an id tag, and I could put paragraphs at the start and end with id tags with no text just for the purpose of getting there position. Is there a property or method that can report what the vertical position of elements WOULD BE (identified by id tags) without actually scrolling to them in the process?

 

Also, can I make an onclick="myfunction()" attribute inside a <p> so that anytime anyone clicks on that paragraph, I can call a function that will set the currentTime() of the audio to the position that starts reading that text? That would be great.

 

Thanks.

 

P.S.

For example, can I use the HTML DOM scrollHeight property? Maybe I can put the entire scrollable content inside a <div id="allStuff"> and get the entire size of the displayable content in pixels even if someone changes the text size or window size or both. Then maybe I can put the audibly read stuff in a <div id="readStuff"> and get the size of only the stuff read out loud. Together with window.innerHeght and window.pageYOffset I can manage to keep the text being read in the middle of the page.

Edited by MikeatW3S

Share this post


Link to post
Share on other sites

Is there a property or method that can report what the vertical position of elements WOULD BE (identified by id tags) without actually scrolling to them in the process?

You can get the current position of any element regardless of whether or not it is visible. The methods vary a little by browser, but if you look that up you should be able to find information about it.Otherwise, it sounds like the best solution is to know the time position of each paragraph in the audio. You would periodically check to see the current time of the audio, figure out which paragraph that is, and scroll there. If they scroll the window then you'll need to figure out which paragraph is near the top or center, get the start audio position of that paragraph, and seek the audio to that point.
1 person likes this

Share this post


Link to post
Share on other sites

So I will need to know the vertical position of certain paragraphs in the document and the time in the audio file where they are read. But I'd like to avoid keeping a table or array of vertical positions of all of the paragraphs with corresponding audio times, since any change of text size and/or window resizing will require a recalculation of the entire table of paragraph locations. I now know how to calculate the vertical position of a <p id="paragraph 1"> by id (or I can make that a <div id="paragraph 1"></div> if needed). So if I know the vertical position of the present paragraph that's being read, can I dynamically go out on the fly and find the next <p id="paragraph 2"> that will be farther down the document? Then I don't have to keep an array of paragraph locations. And I don't even have to know how many there are. I just process until there are no more paragraphs.

 

Also, can I make a <p id="label">, where label is some number, say 1429? Then I can make the label the time marker itself. And when I find the next <p> tag, I will automatically have the time in the audio file where it starts to be read out. This also eliminates the need for a table of document locations and audio locations for each paragraph.

 

Thank you for all your help in this project.

 

P.S. Perhaps I can .getElementById("p1") and label each paragraph with id="p1", id="p2", id="p3", ... Then I might give each paragraph with the additional name="time", where "time" is the numeric value of the time where that paragraph is read. Is this doable? Thanks.

Edited by MikeatW3S

Share this post


Link to post
Share on other sites

You don't need to store the positions, just the IDs of the paragraphs. You can determine the position of each paragraph in run-time. It wouldn't help to store the positions because browser sizes and rendering can change, so store the element IDs and then figure out what the position is when you need to scroll to it.

can I dynamically go out on the fly and find the next <p id="paragraph 2"> that will be farther down the document?

You can use the nextSibling property of an element to get the one that comes after it. That only helps with synching the text with the audio though. If you want to be able to seek through the audio and jump to the appropriate position in the text then you need an array of paragraph IDs and associated audio times. I don't see another way to do it, you need those pairs of data. You could store the audio time as a data attribute of each paragraph instead of an array, but still, you need those pairs of data for it to be useful. e.g.:<p id="p1" data-time="00:40">
1 person likes this

Share this post


Link to post
Share on other sites

Yea, that sounds like the way to go. Can I do something like <p id=2 data-time=00:40></p>? I'm concerned about mismatching data types between strings and numbers, again. And I heard that I don't necessarily need to use quotes on my attributes.

 

I'm also thinking of something like <p id=2 data-time=00:40 onclick="setAudioTiming(00:40)"></p> and when someone clicks on that paragraph, I go to a function that sets the currentTime() of the audio to data-time. Will the onclick attribute work in <p>?

 

Of course, if someone grabs the vertical scroll bar, I'll have to do some more serious calculations to adjust the audio time. But that doesn't sound like too much trouble.

 

Thanks again for the help.

Share this post


Link to post
Share on other sites

I'm concerned about mismatching data types between strings and numbers, again.

All attribute values are strings, you should always quote them. If you expect a number then convert it in Javascript. HTML does not have data types.

And I heard that I don't necessarily need to use quotes on my attributes.

http://stackoverflow.com/questions/6495310/do-you-quote-html5-attributes

Will the onclick attribute work in <p>?

The best thing to do is test it and find out. It would be more flexible to pass this to a click handler, which will pass the entire element that was clicked on. You can get the attributes from the element.
1 person likes this

Share this post


Link to post
Share on other sites

I'm thinking of using:

 

var x = document.getElementById("item1").nextElementSibling

 

to advance which paragraph to display. The trouble is that you have to call the next element by a function of what the present element is, which assumes you know that. I suppose I could relabel the item by which you call the next. I know how to call by id, but how do you get the id label from nextElementSibling? Any help is appreciated.

Share this post


Link to post
Share on other sites

You can always assign the element to a variable that can change.

 

Here's an example using that technique:

var element = document.getElementById("item1");while(element.nextElementSibling) {    // Do something with the element    // For example:    element.style.color = "red";    // Next element:    element = element.nextElementSibling;}
1 person likes this

Share this post


Link to post
Share on other sites

 

You can always assign the element to a variable that can change.

 

Here's an example using that technique:

var element = document.getElementById("item1");while(element.nextElementSibling) {    // Do something with the element    element = element.nextElementSibling;}

 

Thanks. I'll give this a try and let you know.

 

Why should the following work:

 

while(element.nextElementSibling) {

 

Does element.nextElementSibling evaluate to "true" or a number? Does the while(condition) work as long as condition is defined? Or was that not your point in your post? Thanks.

 

PS. element = element.nextElementSibling does not seem to work.

 

PSS. But this does work. I tried it:

 

var indexp = 0; function GetYPosition() { var pelmnt = document.getElementById(indexp); indexp = indexp + 1;

...

}

 

 

It works with paragraph tags labeled as:

 

<p id = 0 >

Some text

</p>

<p id = 1 >

Some more text

</p>

<p id = 3 >

And even more text

</p>

Each time I call the function it processes the next paragraph.

Edited by MikeatW3S

Share this post


Link to post
Share on other sites

You probably won't need a while() loop for your scripts. In my post I was just showing that you can have a variable that points to a different element each time but always has a nextElementSibling property and there's no need to know the ID of the sibling to reference it.

 

In Javascript everything evaluates to true except for zero, null, an empty string and "undefined". If there is no element following the current one, then nextElementSibling will be null or undefined (I forget which) so it evaluates to false.

1 person likes this

Share this post


Link to post
Share on other sites

OK, I can scroll to various paragraphs and get a data-time corresponding to the its time in the audio. Now I'm ready to start writing code to actually scroll the screen slowly. And I have a couple of questions.

 

All the automatic scrolling routines that I've seen rely on the setTimer() function. So I wonder if I set the timer to service a function too fast will this cause too much processing so that the fan turns on? Is there an optimum timer interval between scroll adjustments? Also, if I want to scroll very slowly, can I scroll by fractional steps such as by 0.1? Or must I scrollBy() using whole numbers? As always, any help is appreciated.

Share this post


Link to post
Share on other sites

There's no reason to animate things faster than about once every 35 milliseconds, people won't be able to tell the difference if it's faster. You can't scroll by less than 1 pixel at a time.

Share this post


Link to post
Share on other sites

It seems different browsers scroll at different speeds than others. I suppose I could measure at run time how fast things are scrolling. Or maybe it's possible to find out what browser I'm in. How would I get that information? Thanks.

 

PS.

Or it may be that I had too many tabs open and this affected how the timer routine was serviced. I found that the scroll seems smoother if I set the time interval to 1ms and then count 20 of them before scrolling by one pixel. That way it seems that any problems get averaged out.

 

PSS.

Now I'm concerned that scrolling itself will interfere with calculations of paragraph positions. If I calculate at run time the position of the next paragraph (in order to position it at the correct time), then the window may scroll by a few pixels between getting the position of the scroll bar and getting the position of the paragraph with respect to the viewable window. This may throw off the calculation by a number of pixels. Perhaps the first thing I need to do after I load a page and before scrolling is to make a table at run time of how many paragraphs there are and their positions. Can I create an array that stops adding member when there are no more? How does that work?

Edited by MikeatW3S

Share this post


Link to post
Share on other sites

Hi guys,

 

I seem to have accomplished what I was looking for. I have questions about cleaning up the code. What goes where, when or if to link a js file instead of all the code on every page,. My questions are:

 

1) If I put different sections of javascript code between different <script>...</script> tags, will the various code sections access each others functions and variable? Or must each section be selfcontained? I'd like to put some of the page dependent code in one section and link to a js file with page independent code for the rest. Is that doable?

 

2) When does code in various places run? If I have code where global variables are declared (what's that called - global area?) does that run as soon as the browser gets that portion of code from the server? Or does it start to run only after all the code is loaded or after all the page is loaded? I already know about when onload = "myFunction()" runs and when onunload="myOtherFunction()" runs. I don't know when global code runs.

 

3) Where should code usually go, in the <head> section or in the <body> section?

 

4) What happens with timers when you leave a page? Do they continue to run, or are they automatically shut down?

 

5) What is common programming practice, to declare global variables that are used amongst the various functions, or to pass local variables to functions that have variables in there definition?

 

Thanks again for your help.

Share this post


Link to post
Share on other sites

If I put different sections of javascript code between different <script>...</script> tags, will the various code sections access each others functions and variable?

All of the code will execute in the same scope, the global scope.

When does code in various places run?

As soon as the browser gets to it. When the browser is parsing the HTML code and gets to a script element, it will execute the script at that point before going on with the rest of the HTML.

Where should code usually go, in the <head> section or in the <body> section?

It depends on personal preference. Some people like to put everything in the head, other people like to put everything before the closing body tag. When code is before the closing body tag you typically don't need to use onload handlers, for example, and the scripts at the end won't cause the browser to pause while they are downloaded before the rest of the page loads.

What happens with timers when you leave a page?

When a page is unloaded all Javascript code for the page ends.

What is common programming practice, to declare global variables that are used amongst the various functions, or to pass local variables to functions that have variables in there definition?

It's always preferable to avoid using as many global variables as you can. It's common to define objects that include all of the properties that you would otherwise have as global variables, and to make sure that all of your code is running in the correct scope to access everything.
1 person likes this

Share this post


Link to post
Share on other sites

<thanks>, More Human Than Human,

 

That really helps. If you don't mind, I have a few more questions that are probably easy for you.

 

1) Is it common practice to include on every page all the javascript (js) code that's common to every page? Or should I put this common code in a separate file and "link" to it on every page where it is used. If I link to it, what statement do I put on each page that will fetch the common js code? Do I put the common js file in the root directory or in a separate js directory?

 

2) All my code is for the purpose of synchronizing audio and scrolling, and cookies are used to pass user preferences (to play or not) from one page to the next. What should I do if a user's browser does not support HTML5 or if their cookies are disabled? Can I put some code on each page that downloads the common js code only if the browser supports HTML5 and cookies are enabled? What statements would do that?

 

</thanks>

Edited by MikeatW3S

Share this post


Link to post
Share on other sites

If you have code that needs to run on multiple pages it's best to put it in a separate file and include it on each page. That way the browser can cache it so that it doesn't need to keep downloading the same thing. You can give the URL of the script file inside a script tag.

What should I do if a user's browser does not support HTML5 or if their cookies are disabled?

If you can detect that then you can just show an error message saying what the problem is.

Can I put some code on each page that downloads the common js code only if the browser supports HTML5 and cookies are enabled?

I suppose you could try to detect the browser's capabilities with Javascript and then use Javascript to attach the additional files by creating script elements and appending them to the page.

Share this post


Link to post
Share on other sites

All of the code will execute in the same scope, the global scope.As soon as the browser gets to it. When the browser is parsing the HTML code and gets to a script element, it will execute the script at that point before going on with the rest of the HTML.

 

Does this mean that you must have all the variable declared above the statements that will use them so that when the code is first seen the variables will be meaningful? Or must all the script be downloaded first from <script> to </script> before code is run?

Share this post


Link to post
Share on other sites

The code runs as soon as the browser reaches that point in the file, it doesn't download everything and then execute it. If you're using a particular variable then it does need to be defined in that scope.

1 person likes this

Share this post


Link to post
Share on other sites

If you have code that needs to run on multiple pages it's best to put it in a separate file and include it on each page. That way the browser can cache it so that it doesn't need to keep downloading the same thing. You can give the URL of the script file inside a script tag.

 

I have the page being refreshed from the server each time it is accessed. Otherwise it takes two button clicks to start audio if it is arrived at by a back or forward or refresh button. The first button click restarts the code and places the page at the top. Then the second click starts playing audio and scrolling. So I wonder if I would be downloading a new copy of the separate js file with every access of the page or with every press of the refresh button? Is this right?

Share this post


Link to post
Share on other sites

The code runs as soon as the browser reaches that point in the file, it doesn't download everything and then execute it. If you're using a particular variable then it does need to be defined in that scope.

 

Does this mean that the code on each page should not refer to the separate js script before the js file loads? Do I have to somehow stop execution of code until the js file loads? When does the js file load, when it is called for in a page? Or does it load after it is done loading the present page? Thanks again.

Edited by MikeatW3S

Share this post


Link to post
Share on other sites

So I wonder if I would be downloading a new copy of the separate js file with every access of the page or with every press of the refresh button? Is this right?

You can see what the browser is doing in the developer tools. The network tab will show all of the requests that go out, you can use that to see when it sends the request for the Javascript file.

Does this mean that the code on each page should not refer to the separate js script before the js file loads?

If you're trying to do something like execute a function that is defined in the other file, before that file loads, then it will be a runtime error. If you want to make sure that everything is loaded before you run any code then you can use the body element's onload handler to run a function after everything finishes loading.

Share this post


Link to post
Share on other sites

I'm wondering, since javascript is interpreted by the user's browser at run time and not first compiled, should variables and functions be given the shortest possible names? Should I go through the code and able variable as v1, v2, v3, etc., and functions as f1, f2, f3, etc? Would this make the javascript run faster?

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now
Sign in to follow this  
Followers 0