wget url html problem

[paulm]

Viewing Page Source, and downloading url with wget both show this Google Trends page without the "Stories Trending Now" list, that is visible in Developer Console:

md-list.md-list-block.ng-scope

Searching on this topic points to JavaScript hiding elements of DOM. So how to download this section of the page? Thanks in advance for guidance.

Edited August 7, 2016 by paulm

[paulm]

Just happened on phantomjs, which seems the right direction to get html obfuscated by JavaScript. Phantomjs script is working with w3schools.invisionzone.com as url, but not with Google Trends url. I'm showing the output of the troubleshooting script here; thanks for help.

"headers": [
        {
            "name": "User-Agent",
            "value": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.111 Safari/537.36"
        },
        {
            "name": "Accept",
            "value": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"
        }
    ],
    "id": 1,
    "method": "GET",
    "time": "2016-08-07T14:02:34.247Z",
    "url": "http://www.google.com/trends/home/m/US"
}Request {
    "headers": [
        {
            "name": "User-Agent",
            "value": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.111 Safari/537.36"
        },
        {
            "name": "Accept",
            "value": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"
        }
    ],
    "id": 2,
    "method": "GET",
    "time": "2016-08-07T14:02:34.473Z",
    "url": "https://www.google.com/trends/home/m/US"
}Unable to access network

Edited August 7, 2016 by paulm

[Ingolme]

Any content generated by Javascript will not be in the page's source code, but will be in the DOM. The source code only shows exactly the data that was received from the server.

[paulm]

Right.

[paulm]

I got phantomjs working to retrieve html from https urls by:

path-to/phantom.js --ssl-protocol=any /path-to/script.js

in case it helps someone.

[justsomeguy]

Just to be clear, the HTML is not being hidden or obfuscated, it is being created by Javascript.