Jump to content
iwato

Conquering Direct Access to the Matomo Web-Appliecation without Compromising Security

Recommended Posts

I would like to share with you a message that I posted on the Matomo forum for which I am not expecting a satisfactory reply.  I suspect that Matomo will be unwilling to entertain the idea for fear of opening a Pandora's box of insecurity for casual users and thereby endangering its reputation as a secure web utility.  Matomo's fear need not be my own, however, with the proper guidance.

BACKGROUND:  After several weeks of enormous frustration I was finally able to access the Matomo web application directly with PHP.  I was able to achieve this somewhat (for me) monumental task by renaming the .htaccess file (see below) in the /matomo/config folder and thus canceling its prohibitive effect.

Obviously, this file was placed in the /matomo/config folder as a matter of security -- security, mind you, that I do not wish to compromise.  This said, an unending series of HTTP requests uses up an enormous number of resources, and I wish to share the data that I collect about my users with my users via my own custom-built PHP files for which I require direct communication with the Matomo application. (I have no desire to play the governmental game of protect-and-deceive -- read GDPR or NSA).  

So, I explored the internet and found alternative contents <https://www.slicewise.net/php/piwik-absichern/> to the current .htaccess file in use by Matomo as of 3.5.1.

REQUEST:  Would anyone like to comment on the merging of these two files (see below) into a workable arrangement that would give me the superuser direct access to the Matomo web-application via PHP, but deny access to everyone else except through those channels already set in place by Matomo for anonymous users and opt-in/out visitors.

PROPOSED FILE

<Files "*">
    AuthUserFile /path/to/piwik.htpasswd
    AuthName Piwik-Password
    AuthType Basic
    <RequireAny>
        Require valid-user
        Require ip 127.0.0.1 <your-server-ip> #needed for some scripts
    </RequireAny>
</Files>
<Files ~ "^piwik\.(js|php)|robots\.txt|idxsec\.php|favicon\.ico$">
    Require all granted
</Files>

CURRENT FILE

<Files "*">
<IfModule mod_version.c>
    <IfVersion < 2.4>
        Order Deny,Allow
        Deny from All
    </IfVersion>
    <IfVersion >= 2.4>
        Require all denied
    </IfVersion>
</IfModule>
<IfModule !mod_version.c>
    <IfModule !mod_authz_core.c>
        Order Deny,Allow
        Deny from All
    </IfModule>
    <IfModule mod_authz_core.c>
        Require all denied
    </IfModule>
</IfModule>
</Files>

My goal is to give my visitors an important glimpse into the reality of data collection and analysis.  I now know that I am able to do this my eliminating the .htaccess file.  Elimination of the file is, however, not my goal.

Roddy

Share this post


Link to post
Share on other sites

This is quite a large project you're asking about. I'd expect around 8 hours just for the research and analysis phase. I'd need to get a good understanding of how Matomo works and how exactly it is set up on your website.

This might help you to start off: https://matomo.org/docs/integration/

As for the GDPR and the NSA, they are basically polar opposites. The former is a new regulation by the European Union designed to force large companies to stop harvesting and selling user data carelessly. The latter is a USA organization dedicated to stealing peoples' information regardless of how ethical the methods are. You really should do some research before jumping to conclusions.

Share this post


Link to post
Share on other sites

I have already spent several days.  I know enough to know that something is possible, but in this case do not know enough to do it myself.  Security has rarely been an issue for me, because up until now my livelihood has not depended on it.

I humbly disagree with you about the GDPR and the NSA.  I know of know government that does not wear two faces:  one of benevolence, and the other of power, control,  manipulation, and self-interest.  The GDPR is a show of unneeded protective benevolence to cover for what is truly going on behind the scenes.  You are naïve to think otherwise.

If the EU cared, then it would educate the European public about the reality of data collection and not interfere with private enterprise.  There are enough laws on the books already to handle the problem of identity theft and the abuse of private information.

Roddy

Share this post


Link to post
Share on other sites

Yes, it is difficult, but you either have to learn how to do it yourself by reading books and taking software development courses or pay somebody to do it for you. The work necessary to meet your requirements is much more than could be feasibly communicated through a forum.

Share this post


Link to post
Share on other sites

that would give me the superuser direct access to the Matomo web-application via PHP, but deny access to everyone else except through those channels already set in place by Matomo for anonymous users and opt-in/out visitors.

Are you suggesting that this company you're using has some sort of administration web application on your server, but it is not password-protected?  Like, there's no login page?

Share this post


Link to post
Share on other sites

Matomo allows three channels of external access to the Matomo program:  piwik.php (644), piwik.js (777), and the bot request text file whose name I do not remember.  Each of these channels is accessed via an HTTP request (secure or unsecure, with the option to enforce security). The piwik.js channel is for webpage data collection.  The piwik.php channel is used for data reporting.

The superuser who logs in with his name and password has the authorization to permit anonymous users unlimited access to Matomo's reporting API (read piwik.php).  He can also specify named users.  Whether an anonymous or named user the privileges are the same -- unlimited use of the reporting API.  I do not wish to offer unlimited use, rather, restricted use.  Unfortunately, within the current configuration I have no way of granting restricted autonomous use.  It is nothing or everything for everyone or the same for a select number of users.

All of the critical user information is located in a folder called config.  This folder is a sibling with all other Matomo files and folders located in the Matomo package. Inside the config folder is a .htaccess file.  Eliminating this folder provides me with the access needed to manipulate the files and folders of Matomo without having to make HTTP requests.  In effect, the Matomo package and the Grammar Captive website are located in different branches (add-on domains) of the same root directory.

It seems obvious that the just mentioned .htaccess file is critical to integrity of the Matomo package.  It is the guard dog to the entire PHP package. I need to set the biscuit that will silence the dog as I pass in and out, else I will have to perform all of my commands through HTTP requests, and even is already causing trouble as can be seen through another W3Schools posting.

Roddy

 

Share this post


Link to post
Share on other sites

If you're running PHP on your web server through your browser, then you're using an HTTP request to do that.  I'm not sure what you're talking about with not having to send requests to interact with PHP.  Like, this statement:

Eliminating this folder provides me with the access needed to manipulate the files and folders of Matomo without having to make HTTP requests.

The .htaccess file only deals with requests.  If you are not using an HTTP request, the .htaccess is not even read, it would have no effect.  I'm not sure how you plan to use this web application without sending requests to the server, but if you're not sending requests then .htaccess does not come into play.  It is part of Apache.

Unfortunately, within the current configuration I have no way of granting restricted autonomous use.  It is nothing or everything for everyone or the same for a select number of users.

That sounds like a feature limitation of the software you've chosen to use.

It seems obvious that the just mentioned .htaccess file is critical to integrity of the Matomo package.  It is the guard dog to the entire PHP package.

Well, .htaccess provides additional rules or settings for Apache when serving files under a directory that contains a .htaccess file (anywhere in the directory tree).  So, if this software expects Apache to perform a certain way, then I would say you need to leave that file unchanged.  I'm not sure what the real issue is, with that file there are you unable to access the site?  Is everyone able to access it?

 

Share this post


Link to post
Share on other sites
On 5/30/2018 at 11:51 AM, justsomeguy said:

If you're running PHP on your web server through your browser, then you're using an HTTP request to do that.  I'm not sure what you're talking about with not having to send requests to interact with PHP.

 

Although more complex than what I am writing here, one may think of Matomo, at least for our purposes, as divided into two parts: tracking and reporting.  Although one may track with PHP, I am using Javascript for this purpose.  On the reporting side Matomo offered either of two means:  indirect HTTP requests with a GET query string, or direct access via a large variety of get methods categorize by the PHP classes that defined them.  As Matomo is known on the internet for being resource intensive, and as I am working on shared SSL server, I prefer the latter means of accessing the report methods.  In this spirit I brought it to the attention of Matomo that their recommended code did not perform as suggested, and they have since removed it from their site.  What they did not do is replace it.  Thus, I am left to achieve on my own what appears to be the most efficient manner, or resign myself to what now appears to be prohibited.

I have since been told, by the way, that indirect access via HTTP requests is common procedure.  Considering, however, that the Matomo application and my web application both lie on the same server in the same document tree -- albeit under different add-on domain names --, it seems unreasonable to have to make HTTP requests to achieve my reporting goals.  In the Matomo forum set up for Matomo users Matomo appears to have adopted a stance of silence on this matter.  It is for this reason that I have taken it up with W3Schools.

Roddy

Share this post


Link to post
Share on other sites
Quote

On the reporting side Matomo offered either of two means:  indirect HTTP requests with a GET query string, or direct access via a large variety of get methods categorize by the PHP classes that defined them.

I'm going to assume that you're describing a PHP API.  It's important that everyone uses the same terminology so we know what each other is talking about, but it sounds like you're describing a PHP API that this service provides, like one of these.  But, assuming that all of your data is stored on their servers instead of in a local database or something, even if you're using the API it's still going to send a request to them to record whatever you're doing.  Or, if you're getting data, the API is still going to send a request to their systems for that data.  If the data is stored on their systems you have to send a request, this is not a request-less setup if you're getting data from their system.  The request is just being sent a different way.

Quote

Considering, however, that the Matomo application and my web application both lie on the same server in the same document tree -- albeit under different add-on domain names --, it seems unreasonable to have to make HTTP requests to achieve my reporting goals.

Why?  All of your data is sent to their system and stored there, right?  Why exactly do you think it is "unreasonable" to send a request if you need data from a third-party system?  Is there any alternative that you're aware of, considering that quantum computing is still a ways off?

Since I've noticed it before here, consider that a barrier to this might be terminology - there might already be solutions that you aren't noticing because you don't know the terminology, and people might not understand what your issue is if you're using terminology that they don't normally use.  Every experienced programmer knows what an API is, for example, it doesn't require a definition.  If you're using that, but describing it has a large variety of methods categorized by the PHP classes that defined them, people might be confused that you're just using their API and they might not understand your question because they're having a hard time getting past the terminology barrier.

Share this post


Link to post
Share on other sites

No, you have the wrong idea about Matomo.  I have nicknamed it Google Analytics without Google, for unlike Google everything is yours:  the data that you collect as well as the application itself reside on your own server.  Now, Matomo does offer a cloud, but you are by no means obligated to use it.  Further, Matomo is a very portable self-contained package.  Even, it has its own vendor folder for dependencies.  Once you have downloaded and installed the package, you are free to do with it what you please within the normal restrictions of open-source software. 

Yes, there is an API.  In fact, in the manual the word API is used everywhere to the point where it no longer makes any sense.  In any case, I have been told that all reporting requests must pass through a file called index.php.  With this understanding I composed a .php file with the following contents and ran the same query string that I successfully ran using an HTTP request from another file on my website..  The variable dump produces boolean false.

	ini_set('log_errors', 1);
	ini_set('error_log', dirname(__FILE__) . DIRECTORY_SEPARATOR . 'error.log');
	ini_set('html_errors', 0);
	ini_set('display_errors', 0);
	error_reporting(E_ALL);

	define('PIWIK_INCLUDE_PATH', realpath('../../../../the_path_to_the_matomo_folder/matomo'));
	$result = file_get_contents(PIWIK_INCLUDE_PATH . DIRECTORY_SEPARATOR . 'index.php?module=API&action=index&method=VisitsSummary.get&idSite=1&period=year&date=today&format=json&token_auth=&token_auth=my_authorization_token');
	var_dump($result);

When I examine the error.log file I am told

[02-Jun-2018 01:32:25 UTC] PHP Warning:  file_get_contents(/home/.../public_html/path_to_matomo_folder/php/matomo/index.php?module=API&action=index&method=VisitsSummary.get&idSite=1&period=year&date=today&format=json&token_auth=&token_auth=my_authorization_token): failed to open stream: No such file or directory in /home/.../public_html/path_to the _folder_containing_the_requesting_file/direct_request.php on line 9

Line 9 is file_get_contents() method containing the path to the index.php file.

Roddy

Share this post


Link to post
Share on other sites

You can't pass a query string in the file system. The whole "?module=API&action=index&method=VisitsSummary.get&..." part can only be done through HTTP because that's part of a URL, not a file path.

Share this post


Link to post
Share on other sites

OK.  This makes sense, and a little research has revealed the file that handles the parsed query string.  Obviously I could by-pass the need to write a query string by simply supplying the appropriate values for the public functions that generate the desired reports.   It would appear that my next challenge would be to make sure that everything that is needed is present when I call the appropriate method.  Below is the file in which the appropriate functions are found.  Does a quick review of the file suggest that my idea is plausible?

INSIDE MATOMO  <./matomo/core/FrontController.php>

namespace Piwik;

use Exception;
use Piwik\API\Request;
use Piwik\Container\StaticContainer;
use Piwik\Exception\AuthenticationFailedException;
<?php
/**
 * Piwik - free/libre analytics platform
 *
 * @link http://piwik.org
 * @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
 *
 */

namespace Piwik;

use Exception;
use Piwik\API\Request;
use Piwik\Container\StaticContainer;
use Piwik\Exception\AuthenticationFailedException;
use Piwik\Exception\DatabaseSchemaIsNewerThanCodebaseException;
use Piwik\Exception\PluginDeactivatedException;
use Piwik\Exception\StylesheetLessCompileException;
use Piwik\Http\ControllerResolver;
use Piwik\Http\Router;
use Piwik\Plugins\CoreAdminHome\CustomLogo;

/**
 * This singleton dispatches requests to the appropriate plugin Controller.
 *
 * Piwik uses this class for all requests that go through **index.php**. Plugins can
 * use it to call controller actions of other plugins.
 *
 * ### Examples
 *
 * **Forwarding controller requests**
 *
 *     public function myConfiguredRealtimeMap()
 *     {
 *         $_GET['changeVisitAlpha'] = false;
 *         $_GET['removeOldVisits'] = false;
 *         $_GET['showFooterMessage'] = false;
 *         return FrontController::getInstance()->dispatch('UserCountryMap', 'realtimeMap');
 *     }
 *
 * **Using other plugin controller actions**
 *
 *     public function myPopupWithRealtimeMap()
 *     {
 *         $_GET['changeVisitAlpha'] = false;
 *         $_GET['removeOldVisits'] = false;
 *         $_GET['showFooterMessage'] = false;
 *         $realtimeMap = FrontController::getInstance()->dispatch('UserCountryMap', 'realtimeMap');
 *
 *         $view = new View('@MyPlugin/myPopupWithRealtimeMap.twig');
 *         $view->realtimeMap = $realtimeMap;
 *         return $realtimeMap->render();
 *     }
 *
 * For a detailed explanation, see the documentation [here](https://developer.piwik.org/guides/how-piwik-works).
 *
 * @method static \Piwik\FrontController getInstance()
 */
class FrontController extends Singleton
{
    const DEFAULT_MODULE = 'CoreHome';

    /**
     * Set to false and the Front Controller will not dispatch the request
     *
     * @var bool
     */
    public static $enableDispatch = true;

    /**
     * @var bool
     */
    private $initialized = false;

    /**
     * @param $lastError
     * @return string
     * @throws AuthenticationFailedException
     * @throws Exception
     */
    private static function generateSafeModeOutputFromError($lastError)
    {
        Common::sendResponseCode(500);

        $controller = FrontController::getInstance();
        try {
            $controller->init();
            $message = $controller->dispatch('CorePluginsAdmin', 'safemode', array($lastError));
        } catch(Exception $e) {
            // may fail in safe mode (eg. global.ini.php not found)
            $message = sprintf("Matomo encoutered an error: %s (which lead to: %s)", $lastError['message'], $e->getMessage());
        }

        return $message;
    }

    /**
     * @param Exception $e
     * @return string
     */
    private static function generateSafeModeOutputFromException($e)
    {
        $error = array(
            'message' => $e->getMessage(),
            'file' => $e->getFile(),
            'line' => $e->getLine()
        );
        return self::generateSafeModeOutputFromError($error);
    }

    /**
     * Executes the requested plugin controller method.
     *
     * @throws Exception|\Piwik\Exception\PluginDeactivatedException in case the plugin doesn't exist, the action doesn't exist,
     *                                                     there is not enough permission, etc.
     *
     * @param string $module The name of the plugin whose controller to execute, eg, `'UserCountryMap'`.
     * @param string $action The controller method name, eg, `'realtimeMap'`.
     * @param array $parameters Array of parameters to pass to the controller method.
     * @return void|mixed The returned value of the call. This is the output of the controller method.
     * @api
     */
    public function dispatch($module = null, $action = null, $parameters = null)
    {
        if (self::$enableDispatch === false) {
            return;
        }

        $filter = new Router();
        $redirection = $filter->filterUrl(Url::getCurrentUrl());
        if ($redirection !== null) {
            Url::redirectToUrl($redirection);
            return;
        }

        try {
            $result = $this->doDispatch($module, $action, $parameters);
            return $result;
        } catch (NoAccessException $exception) {
            Log::debug($exception);

            /**
             * Triggered when a user with insufficient access permissions tries to view some resource.
             *
             * This event can be used to customize the error that occurs when a user is denied access
             * (for example, displaying an error message, redirecting to a page other than login, etc.).
             *
             * @param \Piwik\NoAccessException $exception The exception that was caught.
             */
            Piwik::postEvent('User.isNotAuthorized', array($exception), $pending = true);
        } catch (\Twig_Error_Runtime $e) {
            echo $this->generateSafeModeOutputFromException($e);
            exit;
        } catch(StylesheetLessCompileException $e) {
            echo $this->generateSafeModeOutputFromException($e);
            exit;
        } catch(\Error $e) {
            echo $this->generateSafeModeOutputFromException($e);
            exit;
        }
    }

    /**
     * Executes the requested plugin controller method and returns the data, capturing anything the
     * method `echo`s.
     *
     * _Note: If the plugin controller returns something, the return value is returned instead
     * of whatever is in the output buffer._
     *
     * @param string $module The name of the plugin whose controller to execute, eg, `'UserCountryMap'`.
     * @param string $actionName The controller action name, eg, `'realtimeMap'`.
     * @param array $parameters Array of parameters to pass to the controller action method.
     * @return string The `echo`'d data or the return value of the controller action.
     * @deprecated
     */
    public function fetchDispatch($module = null, $actionName = null, $parameters = null)
    {
        ob_start();
        $output = $this->dispatch($module, $actionName, $parameters);
        // if nothing returned we try to load something that was printed on the screen
        if (empty($output)) {
            $output = ob_get_contents();
        } else {
            // if something was returned, flush output buffer as it is meant to be written to the screen
            ob_flush();
        }
        ob_end_clean();
        return $output;
    }

    /**
     * Called at the end of the page generation
     */
    public function __destruct()
    {
        try {
            if (class_exists('Piwik\\Profiler')
                && !SettingsServer::isTrackerApiRequest()
            ) {
                // in tracker mode Piwik\Tracker\Db\Pdo\Mysql does currently not implement profiling
                Profiler::displayDbProfileReport();
                Profiler::printQueryCount();
            }
        } catch (Exception $e) {
            Log::debug($e);
        }
    }

    // Should we show exceptions messages directly rather than display an html error page?
    public static function shouldRethrowException()
    {
        // If we are in no dispatch mode, eg. a script reusing Piwik libs,
        // then we should return the exception directly, rather than trigger the event "bad config file"
        // which load the HTML page of the installer with the error.
        return (defined('PIWIK_ENABLE_DISPATCH') && !PIWIK_ENABLE_DISPATCH)
        || Common::isPhpCliMode()
        || SettingsServer::isArchivePhpTriggered();
    }

    public static function setUpSafeMode()
    {
        register_shutdown_function(array('\\Piwik\\FrontController', 'triggerSafeModeWhenError'));
    }

    public static function triggerSafeModeWhenError()
    {
        $lastError = error_get_last();
        if (!empty($lastError) && $lastError['type'] == E_ERROR) {
            $message = self::generateSafeModeOutputFromError($lastError);
            echo $message;
        }
    }

    /**
     * Must be called before dispatch()
     * - checks that directories are writable,
     * - loads the configuration file,
     * - loads the plugin,
     * - inits the DB connection,
     * - etc.
     *
     * @throws Exception
     * @return void
     */
    public function init()
    {
        if ($this->initialized) {
            return;
        }

        $this->initialized = true;

        $tmpPath = StaticContainer::get('path.tmp');

        $directoriesToCheck = array(
            $tmpPath,
            $tmpPath . '/assets/',
            $tmpPath . '/cache/',
            $tmpPath . '/logs/',
            $tmpPath . '/tcpdf/',
            $tmpPath . '/templates_c/',
        );

        Filechecks::dieIfDirectoriesNotWritable($directoriesToCheck);

        $this->handleMaintenanceMode();
        $this->handleProfiler();
        $this->handleSSLRedirection();

        Plugin\Manager::getInstance()->loadPluginTranslations();
        Plugin\Manager::getInstance()->loadActivatedPlugins();

        // try to connect to the database
        try {
            Db::createDatabaseObject();
            Db::fetchAll("SELECT DATABASE()");
        } catch (Exception $exception) {
            if (self::shouldRethrowException()) {
                throw $exception;
            }

            Log::debug($exception);

            /**
             * Triggered when Piwik cannot connect to the database.
             *
             * This event can be used to start the installation process or to display a custom error
             * message.
             *
             * @param Exception $exception The exception thrown from creating and testing the database
             *                             connection.
             */
            Piwik::postEvent('Db.cannotConnectToDb', array($exception), $pending = true);

            throw $exception;
        }

        // try to get an option (to check if data can be queried)
        try {
            Option::get('TestingIfDatabaseConnectionWorked');
        } catch (Exception $exception) {
            if (self::shouldRethrowException()) {
                throw $exception;
            }

            Log::debug($exception);

            /**
             * Triggered when Piwik cannot access database data.
             *
             * This event can be used to start the installation process or to display a custom error
             * message.
             *
             * @param Exception $exception The exception thrown from trying to get an option value.
             */
            Piwik::postEvent('Config.badConfigurationFile', array($exception), $pending = true);

            throw $exception;
        }

        // Init the Access object, so that eg. core/Updates/* can enforce Super User and use some APIs
        Access::getInstance();

        /**
         * Triggered just after the platform is initialized and plugins are loaded.
         *
         * This event can be used to do early initialization.
         *
         * _Note: At this point the user is not authenticated yet._
         */
        Piwik::postEvent('Request.dispatchCoreAndPluginUpdatesScreen');

        $this->throwIfPiwikVersionIsOlderThanDBSchema();

        $module = Piwik::getModule();
        $action = Piwik::getAction();

        if (empty($module)
            || empty($action)
            || $module !== 'Installation'
            || !in_array($action, array('getInstallationCss', 'getInstallationJs'))) {
            \Piwik\Plugin\Manager::getInstance()->installLoadedPlugins();
        }

        // ensure the current Piwik URL is known for later use
        if (method_exists('Piwik\SettingsPiwik', 'getPiwikUrl')) {
            SettingsPiwik::getPiwikUrl();
        }

        /**
         * Triggered before the user is authenticated, when the global authentication object
         * should be created.
         *
         * Plugins that provide their own authentication implementation should use this event
         * to set the global authentication object (which must derive from {@link Piwik\Auth}).
         *
         * **Example**
         *
         *     Piwik::addAction('Request.initAuthenticationObject', function() {
         *         StaticContainer::getContainer()->set('Piwik\Auth', new MyAuthImplementation());
         *     });
         */
        Piwik::postEvent('Request.initAuthenticationObject');
        try {
            $authAdapter = StaticContainer::get('Piwik\Auth');
        } catch (Exception $e) {
            $message = "Authentication object cannot be found in the container. Maybe the Login plugin is not activated?
                        <br />You can activate the plugin by adding:<br />
                        <code>Plugins[] = Login</code><br />
                        under the <code>[Plugins]</code> section in your config/config.ini.php";

            $ex = new AuthenticationFailedException($message);
            $ex->setIsHtmlMessage();

            throw $ex;
        }
        Access::getInstance()->reloadAccess($authAdapter);

        // Force the auth to use the token_auth if specified, so that embed dashboard
        // and all other non widgetized controller methods works fine
        if (Common::getRequestVar('token_auth', false, 'string') !== false) {
            Request::reloadAuthUsingTokenAuth();
        }
        SettingsServer::raiseMemoryLimitIfNecessary();

        \Piwik\Plugin\Manager::getInstance()->postLoadPlugins();

        /**
         * Triggered after the platform is initialized and after the user has been authenticated, but
         * before the platform has handled the request.
         *
         * Piwik uses this event to check for updates to Piwik.
         */
        Piwik::postEvent('Platform.initialized');
    }

    protected function prepareDispatch($module, $action, $parameters)
    {
        if (is_null($module)) {
            $module = Common::getRequestVar('module', self::DEFAULT_MODULE, 'string');
        }

        if (is_null($action)) {
            $action = Common::getRequestVar('action', false);
        }

        if (SettingsPiwik::isPiwikInstalled()
            && ($module !== 'API' || ($action && $action !== 'index'))
        ) {
            Session::start();

            $this->closeSessionEarlyForFasterUI();
        }

        if (is_null($parameters)) {
            $parameters = array();
        }

        if (!ctype_alnum($module)) {
            throw new Exception("Invalid module name '$module'");
        }

        list($module, $action) = Request::getRenamedModuleAndAction($module, $action);

        if (!\Piwik\Plugin\Manager::getInstance()->isPluginActivated($module)) {
            throw new PluginDeactivatedException($module);
        }

        return array($module, $action, $parameters);
    }

    protected function handleMaintenanceMode()
    {
        if ((Config::getInstance()->General['maintenance_mode'] != 1) || Common::isPhpCliMode()) {
            return;
        }
        Common::sendResponseCode(503);

        $logoUrl = 'plugins/Morpheus/images/logo.svg';
        $faviconUrl = 'plugins/CoreHome/images/favicon.png';
        try {
            $logo = new CustomLogo();
            if ($logo->hasSVGLogo()) {
                $logoUrl = $logo->getSVGLogoUrl();
            } else {
                $logoUrl = $logo->getHeaderLogoUrl();
            }
            $faviconUrl = $logo->getPathUserFavicon();
        } catch (Exception $ex) {
        }

        $recordStatistics = Config::getInstance()->Tracker['record_statistics'];
        $trackMessage = '';

        if ($recordStatistics) {
          $trackMessage = 'Your analytics data will continue to be tracked as normal.';
        } else {
          $trackMessage = 'While the maintenance mode is active, data tracking is disabled.';
        }

        $page = file_get_contents(PIWIK_INCLUDE_PATH . '/plugins/Morpheus/templates/maintenance.tpl');
        $page = str_replace('%logoUrl%', $logoUrl, $page);
        $page = str_replace('%faviconUrl%', $faviconUrl, $page);
        $page = str_replace('%piwikTitle%', Piwik::getRandomTitle(), $page);

        $page = str_replace('%trackMessage%', $trackMessage, $page);

        echo $page;
        exit;
    }

    protected function handleSSLRedirection()
    {
        // Specifically disable for the opt out iframe
        if (Piwik::getModule() == 'CoreAdminHome' && Piwik::getAction() == 'optOut') {
            return;
        }
        // Disable Https for VisitorGenerator
        if (Piwik::getModule() == 'VisitorGenerator') {
            return;
        }
        if (Common::isPhpCliMode()) {
            return;
        }
        // proceed only when force_ssl = 1
        if (!SettingsPiwik::isHttpsForced()) {
            return;
        }
        Url::redirectToHttps();
    }

    private function closeSessionEarlyForFasterUI()
    {
        $isDashboardReferrer = !empty($_SERVER['HTTP_REFERER']) && strpos($_SERVER['HTTP_REFERER'], 'module=CoreHome&action=index') !== false;
        $isAllWebsitesReferrer = !empty($_SERVER['HTTP_REFERER']) && strpos($_SERVER['HTTP_REFERER'], 'module=MultiSites&action=index') !== false;

        if ($isDashboardReferrer
            && !empty($_POST['token_auth'])
            && Common::getRequestVar('widget', 0, 'int') === 1
        ) {
            Session::close();
        }

        if (($isDashboardReferrer || $isAllWebsitesReferrer)
            && Common::getRequestVar('viewDataTable', '', 'string') === 'sparkline'
        ) {
            Session::close();
        }
    }

    private function handleProfiler()
    {
        if (!empty($_GET['xhprof'])) {
            $mainRun = $_GET['xhprof'] == 1; // core:archive command sets xhprof=2
            Profiler::setupProfilerXHProf($mainRun);
        }
    }

    /**
     * @param $module
     * @param $action
     * @param $parameters
     * @return mixed
     */
    private function doDispatch($module, $action, $parameters)
    {
        list($module, $action, $parameters) = $this->prepareDispatch($module, $action, $parameters);

        /**
         * Triggered directly before controller actions are dispatched.
         *
         * This event can be used to modify the parameters passed to one or more controller actions
         * and can be used to change the controller action being dispatched to.
         *
         * @param string &$module The name of the plugin being dispatched to.
         * @param string &$action The name of the controller method being dispatched to.
         * @param array &$parameters The arguments passed to the controller action.
         */
        Piwik::postEvent('Request.dispatch', array(&$module, &$action, &$parameters));

        /** @var ControllerResolver $controllerResolver */
        $controllerResolver = StaticContainer::get('Piwik\Http\ControllerResolver');

        $controller = $controllerResolver->getController($module, $action, $parameters);

        /**
         * Triggered directly before controller actions are dispatched.
         *
         * This event exists for convenience and is triggered directly after the {@hook Request.dispatch}
         * event is triggered.
         *
         * It can be used to do the same things as the {@hook Request.dispatch} event, but for one controller
         * action only. Using this event will result in a little less code than {@hook Request.dispatch}.
         *
         * @param array &$parameters The arguments passed to the controller action.
         */
        Piwik::postEvent(sprintf('Controller.%s.%s', $module, $action), array(&$parameters));

        $result = call_user_func_array($controller, $parameters);

        /**
         * Triggered after a controller action is successfully called.
         *
         * This event exists for convenience and is triggered immediately before the {@hook Request.dispatch.end}
         * event is triggered.
         *
         * It can be used to do the same things as the {@hook Request.dispatch.end} event, but for one
         * controller action only. Using this event will result in a little less code than
         * {@hook Request.dispatch.end}.
         *
         * @param mixed &$result The result of the controller action.
         * @param array $parameters The arguments passed to the controller action.
         */
        Piwik::postEvent(sprintf('Controller.%s.%s.end', $module, $action), array(&$result, $parameters));

        /**
         * Triggered after a controller action is successfully called.
         *
         * This event can be used to modify controller action output (if any) before the output is returned.
         *
         * @param mixed &$result The controller action result.
         * @param array $parameters The arguments passed to the controller action.
         */
        Piwik::postEvent('Request.dispatch.end', array(&$result, $module, $action, $parameters));

        return $result;
    }

    /**
     * This method ensures that Piwik Platform cannot be running when using a NEWER database.
     */
    private function throwIfPiwikVersionIsOlderThanDBSchema()
    {
        // When developing this situation happens often when switching branches
        if (Development::isEnabled()) {
            return;
        }

        $updater = new Updater();

        $dbSchemaVersion = $updater->getCurrentComponentVersion('core');
        $current = Version::VERSION;
        if (-1 === version_compare($current, $dbSchemaVersion)) {
            $messages = array(
                Piwik::translate('General_ExceptionDatabaseVersionNewerThanCodebase', array($current, $dbSchemaVersion)),
                Piwik::translate('General_ExceptionDatabaseVersionNewerThanCodebaseWait'),
                // we cannot fill in the Super User emails as we are failing before Authentication was ready
                Piwik::translate('General_ExceptionContactSupportGeneric', array('', ''))
            );
            throw new DatabaseSchemaIsNewerThanCodebaseException(implode(" ", $messages));
        }
    }
}

Roddy

Share this post


Link to post
Share on other sites

If you're trying to do all this without using the HTTP API then I guess you've got your work cut out for you.  There's a class here that is written to provide a PHP API, but this uses the existing HTTP API.  It's 5,000 or so lines of code though.

https://github.com/VisualAppeal/Piwik-PHP-API

I'm not sure why you've decided to spend your time trying to bypass their established API.  If you're going to keep your installation up to date, be prepared for updates to break your code if you're not going to use the prescribed API.

Share this post


Link to post
Share on other sites

Are you suggesting that the class designed to allow direct PHP access to the Matomo server  without HTTP requests, actually uses HTTP requests to communicate with the server, anyway?  If this is true, then I would like to thank you very much for this discovery.  It moves contrary to my whole reason for wanting to avoid having to use HTTP requests -- better performance.  Matomo is well-known on the internet for being resource-intensive, and my host server is already complaining.  I am under tremendous pressure to move up to a VPS cloud,, but I am financially reluctant.  I am just not there yet.

Roddy

Share this post


Link to post
Share on other sites

Are you suggesting that the class designed to allow direct PHP access to the Matomo server  without HTTP requests, actually uses HTTP requests to communicate with the server, anyway?

I'm suggesting that's what the specific project I linked to does.  The only reporting API I see documentation for on the Matomo website uses the HTTP API, I don't see another one.

Share this post


Link to post
Share on other sites

To the best of my knowledge there never was an API per se.  Simply a poorly constructed body of code that did not work.  When I reported its dysfunctionality they took it down, but replaced it with nothing.  Most unfortunately, I was not provided with an alternative body of code.  There was no explanation, only acknowledgement and gratitude for my having reported the dysfunctionality.

Roddy

Share this post


Link to post
Share on other sites

I could find nothing in the file that resembles an HTTP request.  I agree that this is one of the key files for making requests.  There are others as well, however.  Let's go one step at a time.

<?php
/**
 * Piwik - free/libre analytics platform
 *
 * @link http://piwik.org
 * @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later
 *
 */
namespace Piwik\API;

use Exception;
use Piwik\Access;
use Piwik\Cache;
use Piwik\Common;
use Piwik\DataTable;
use Piwik\Exception\PluginDeactivatedException;
use Piwik\IP;
use Piwik\Log;
use Piwik\Piwik;
use Piwik\Plugin\Manager as PluginManager;
use Piwik\Plugins\CoreHome\LoginWhitelist;
use Piwik\SettingsServer;
use Piwik\Url;
use Piwik\UrlHelper;

/**
 * Dispatches API requests to the appropriate API method.
 *
 * The Request class is used throughout Piwik to call API methods. The difference
 * between using Request and calling API methods directly is that Request
 * will do more after calling the API including: applying generic filters, applying queued filters,
 * and handling the **flat** and **label** query parameters.
 *
 * Additionally, the Request class will **forward current query parameters** to the request
 * which is more convenient than calling {@link Piwik\Common::getRequestVar()} many times over.
 *
 * In most cases, using a Request object to query the API is the correct approach.
 *
 * ### Post-processing
 *
 * The return value of API methods undergo some extra processing before being returned by Request.
 *
 * ### Output Formats
 *
 * The value returned by Request will be serialized to a certain format before being returned.
 *
 * ### Examples
 *
 * **Basic Usage**
 *
 *     $request = new Request('method=UserLanguage.getLanguage&idSite=1&date=yesterday&period=week'
 *                          . '&format=xml&filter_limit=5&filter_offset=0')
 *     $result = $request->process();
 *     echo $result;
 *
 * **Getting a unrendered DataTable**
 *
 *     // use the convenience method 'processRequest'
 *     $dataTable = Request::processRequest('UserLanguage.getLanguage', array(
 *         'idSite' => 1,
 *         'date' => 'yesterday',
 *         'period' => 'week',
 *         'filter_limit' => 5,
 *         'filter_offset' => 0
 *
 *         'format' => 'original', // this is the important bit
 *     ));
 *     echo "This DataTable has " . $dataTable->getRowsCount() . " rows.";
 *
 * @see http://piwik.org/docs/analytics-api
 * @api
 */
class Request
{
    private $request = null;

    /**
     * Converts the supplied request string into an array of query paramater name/value
     * mappings. The current query parameters (everything in `$_GET` and `$_POST`) are
     * forwarded to request array before it is returned.
     *
     * @param string|array|null $request The base request string or array, eg,
     *                                   `'module=UserLanguage&action=getLanguage'`.
     * @param array $defaultRequest Default query parameters. If a query parameter is absent in `$request`, it will be loaded
     *                              from this. Defaults to `$_GET + $_POST`.
     * @return array
     */
    public static function getRequestArrayFromString($request, $defaultRequest = null)
    {
        if ($defaultRequest === null) {
            $defaultRequest = self::getDefaultRequest();

            $requestRaw = self::getRequestParametersGET();
            if (!empty($requestRaw['segment'])) {
                $defaultRequest['segment'] = $requestRaw['segment'];
            }

            if (!isset($defaultRequest['format_metrics'])) {
                $defaultRequest['format_metrics'] = 'bc';
            }
        }

        $requestArray = $defaultRequest;

        if (!is_null($request)) {
            if (is_array($request)) {
                $requestParsed = $request;
            } else {
                $request = trim($request);
                $request = str_replace(array("\n", "\t"), '', $request);

                $requestParsed = UrlHelper::getArrayFromQueryString($request);
            }

            $requestArray = $requestParsed + $defaultRequest;
        }

        foreach ($requestArray as &$element) {
            if (!is_array($element)) {
                $element = trim($element);
            }
        }
        return $requestArray;
    }

    /**
     * Constructor.
     *
     * @param string|array $request Query string that defines the API call (must at least contain a **method** parameter),
     *                              eg, `'method=UserLanguage.getLanguage&idSite=1&date=yesterday&period=week&format=xml'`
     *                              If a request is not provided, then we use the values in the `$_GET` and `$_POST`
     *                              superglobals.
     * @param array $defaultRequest Default query parameters. If a query parameter is absent in `$request`, it will be loaded
     *                              from this. Defaults to `$_GET + $_POST`.
     */
    public function __construct($request = null, $defaultRequest = null)
    {
        $this->request = self::getRequestArrayFromString($request, $defaultRequest);
        $this->sanitizeRequest();
        $this->renameModuleAndActionInRequest();
    }

    /**
     * For backward compatibility: Piwik API still works if module=Referers,
     * we rewrite to correct renamed plugin: Referrers
     *
     * @param $module
     * @param $action
     * @return array( $module, $action )
     * @ignore
     */
    public static function getRenamedModuleAndAction($module, $action)
    {
        /**
         * This event is posted in the Request dispatcher and can be used
         * to overwrite the Module and Action to dispatch.
         * This is useful when some Controller methods or API methods have been renamed or moved to another plugin.
         *
         * @param $module string
         * @param $action string
         */
        Piwik::postEvent('Request.getRenamedModuleAndAction', array(&$module, &$action));

        return array($module, $action);
    }

    /**
     * Make sure that the request contains no logical errors
     */
    private function sanitizeRequest()
    {
        // The label filter does not work with expanded=1 because the data table IDs have a different meaning
        // depending on whether the table has been loaded yet. expanded=1 causes all tables to be loaded, which
        // is why the label filter can't descend when a recursive label has been requested.
        // To fix this, we remove the expanded parameter if a label parameter is set.
        if (isset($this->request['label']) && !empty($this->request['label'])
            && isset($this->request['expanded']) && $this->request['expanded']
        ) {
            unset($this->request['expanded']);
        }
    }

    /**
     * Dispatches the API request to the appropriate API method and returns the result
     * after post-processing.
     *
     * Post-processing includes:
     *
     * - flattening if **flat** is 0
     * - running generic filters unless **disable_generic_filters** is set to 1
     * - URL decoding label column values
     * - running queued filters unless **disable_queued_filters** is set to 1
     * - removing columns based on the values of the **hideColumns** and **showColumns** query parameters
     * - filtering rows if the **label** query parameter is set
     * - converting the result to the appropriate format (ie, XML, JSON, etc.)
     *
     * If `'original'` is supplied for the output format, the result is returned as a PHP
     * object.
     *
     * @throws PluginDeactivatedException if the module plugin is not activated.
     * @throws Exception if the requested API method cannot be called, if required parameters for the
     *                   API method are missing or if the API method throws an exception and the **format**
     *                   query parameter is **original**.
     * @return DataTable|Map|string The data resulting from the API call.
     */
    public function process()
    {
        // read the format requested for the output data
        $outputFormat = strtolower(Common::getRequestVar('format', 'xml', 'string', $this->request));

        $disablePostProcessing = $this->shouldDisablePostProcessing();

        // create the response
        $response = new ResponseBuilder($outputFormat, $this->request);
        if ($disablePostProcessing) {
            $response->disableDataTablePostProcessor();
        }

        $corsHandler = new CORSHandler();
        $corsHandler->handle();

        $tokenAuth = Common::getRequestVar('token_auth', '', 'string', $this->request);
        $shouldReloadAuth = false;

        try {

            // IP check is needed here as we cannot listen to API.Request.authenticate as it would then not return proper API format response.
            // We can also not do it by listening to API.Request.dispatch as by then the user is already authenticated and we want to make sure
            // to not expose any information in case the IP is not whitelisted.
            $whitelist = new LoginWhitelist();
            if ($whitelist->shouldCheckWhitelist() && $whitelist->shouldWhitelistApplyToAPI()) {
                $ip = IP::getIpFromHeader();
                $whitelist->checkIsWhitelisted($ip);
            }

            // read parameters
            $moduleMethod = Common::getRequestVar('method', null, 'string', $this->request);

            list($module, $method) = $this->extractModuleAndMethod($moduleMethod);
            list($module, $method) = self::getRenamedModuleAndAction($module, $method);

            PluginManager::getInstance()->checkIsPluginActivated($module);

            $apiClassName = self::getClassNameAPI($module);

            if ($shouldReloadAuth = self::shouldReloadAuthUsingTokenAuth($this->request)) {
                $access = Access::getInstance();
                $tokenAuthToRestore = $access->getTokenAuth();
                $hadSuperUserAccess = $access->hasSuperUserAccess();
                self::forceReloadAuthUsingTokenAuth($tokenAuth);
            }

            // call the method
            $returnedValue = Proxy::getInstance()->call($apiClassName, $method, $this->request);

            $toReturn = $response->getResponse($returnedValue, $module, $method);
        } catch (Exception $e) {
            Log::debug($e);

            $toReturn = $response->getResponseException($e);
        }

        if ($shouldReloadAuth) {
            $this->restoreAuthUsingTokenAuth($tokenAuthToRestore, $hadSuperUserAccess);
        }

        return $toReturn;
    }

    private function restoreAuthUsingTokenAuth($tokenToRestore, $hadSuperUserAccess)
    {
        // if we would not make sure to unset super user access, the tokenAuth would be not authenticated and any
        // token would just keep super user access (eg if the token that was reloaded before had super user access)
        Access::getInstance()->setSuperUserAccess(false);

        // we need to restore by reloading the tokenAuth as some permissions could have been removed in the API
        // request etc. Otherwise we could just store a clone of Access::getInstance() and restore here
        self::forceReloadAuthUsingTokenAuth($tokenToRestore);

        if ($hadSuperUserAccess && !Access::getInstance()->hasSuperUserAccess()) {
            // we are in context of `doAsSuperUser()` and need to restore this behaviour
            Access::getInstance()->setSuperUserAccess(true);
        }
    }

    /**
     * Returns the name of a plugin's API class by plugin name.
     *
     * @param string $plugin The plugin name, eg, `'Referrers'`.
     * @return string The fully qualified API class name, eg, `'\Piwik\Plugins\Referrers\API'`.
     */
    public static function getClassNameAPI($plugin)
    {
        return sprintf('\Piwik\Plugins\%s\API', $plugin);
    }

    /**
     * @ignore
     * @internal
     * @param bool $isRootRequestApiRequest
     */
    public static function setIsRootRequestApiRequest($isRootRequestApiRequest)
    {
        Cache::getTransientCache()->save('API.setIsRootRequestApiRequest', $isRootRequestApiRequest);
    }

    /**
     * Detect if the root request (the actual request) is an API request or not. To detect whether an API is currently
     * request within any request, have a look at {@link isApiRequest()}.
     *
     * @return bool
     * @throws Exception
     */
    public static function isRootRequestApiRequest()
    {
        $isApi = Cache::getTransientCache()->fetch('API.setIsRootRequestApiRequest');
        return !empty($isApi);
    }

    /**
     * Detect if request is an API request. Meaning the module is 'API' and an API method having a valid format was
     * specified. Note that this method will return true even if the actual request is for example a regular UI
     * reporting page request but within this request we are currently processing an API request (eg a
     * controller calls Request::processRequest('API.getMatomoVersion')). To find out if the root request is an API
     * request or not, call {@link isRootRequestApiRequest()}
     *
     * @param array $request  eg array('module' => 'API', 'method' => 'Test.getMethod')
     * @return bool
     * @throws Exception
     */
    public static function isApiRequest($request)
    {
        $module = Common::getRequestVar('module', '', 'string', $request);
        $method = Common::getRequestVar('method', '', 'string', $request);

        return $module === 'API' && !empty($method) && (count(explode('.', $method)) === 2);
    }

    /**
     * If the token_auth is found in the $request parameter,
     * the current session will be authenticated using this token_auth.
     * It will overwrite the previous Auth object.
     *
     * @param array $request If null, uses the default request ($_GET)
     * @return void
     * @ignore
     */
    public static function reloadAuthUsingTokenAuth($request = null)
    {
        // if a token_auth is specified in the API request, we load the right permissions
        $token_auth = Common::getRequestVar('token_auth', '', 'string', $request);

        if (self::shouldReloadAuthUsingTokenAuth($request)) {
            self::forceReloadAuthUsingTokenAuth($token_auth);
        }
    }

    /**
     * The current session will be authenticated using this token_auth.
     * It will overwrite the previous Auth object.
     *
     * @param string $tokenAuth
     * @return void
     */
    private static function forceReloadAuthUsingTokenAuth($tokenAuth)
    {
        /**
         * Triggered when authenticating an API request, but only if the **token_auth**
         * query parameter is found in the request.
         *
         * Plugins that provide authentication capabilities should subscribe to this event
         * and make sure the global authentication object (the object returned by `StaticContainer::get('Piwik\Auth')`)
         * is setup to use `$token_auth` when its `authenticate()` method is executed.
         *
         * @param string $token_auth The value of the **token_auth** query parameter.
         */
        Piwik::postEvent('API.Request.authenticate', array($tokenAuth));
        Access::getInstance()->reloadAccess();
        SettingsServer::raiseMemoryLimitIfNecessary();
    }

    private static function shouldReloadAuthUsingTokenAuth($request)
    {
        if (is_null($request)) {
            $request = self::getDefaultRequest();
        }

        if (!isset($request['token_auth'])) {
            // no token is given so we just keep the current loaded user
            return false;
        }

        // a token is specified, we need to reload auth in case it is different than the current one, even if it is empty
        $tokenAuth = Common::getRequestVar('token_auth', '', 'string', $request);

        // not using !== is on purpose as getTokenAuth() might return null whereas $tokenAuth is '' . In this case
        // we do not need to reload.

        return $tokenAuth != Access::getInstance()->getTokenAuth();
    }

    /**
     * Returns array($class, $method) from the given string $class.$method
     *
     * @param string $parameter
     * @throws Exception
     * @return array
     */
    private function extractModuleAndMethod($parameter)
    {
        $a = explode('.', $parameter);
        if (count($a) != 2) {
            throw new Exception("The method name is invalid. Expected 'module.methodName'");
        }
        return $a;
    }

    /**
     * Helper method that processes an API request in one line using the variables in `$_GET`
     * and `$_POST`.
     *
     * @param string $method The API method to call, ie, `'Actions.getPageTitles'`.
     * @param array $paramOverride The parameter name-value pairs to use instead of what's
     *                             in `$_GET` & `$_POST`.
     * @param array $defaultRequest Default query parameters. If a query parameter is absent in `$request`, it will be loaded
     *                              from this. Defaults to `$_GET + $_POST`.
     *
     *                              To avoid using any parameters from $_GET or $_POST, set this to an empty `array()`.
     * @return mixed The result of the API request. See {@link process()}.
     */
    public static function processRequest($method, $paramOverride = array(), $defaultRequest = null)
    {
        $params = array();
        $params['format'] = 'original';
        $params['serialize'] = '0';
        $params['module'] = 'API';
        $params['method'] = $method;
        $params = $paramOverride + $params;

        // process request
        $request = new Request($params, $defaultRequest);
        return $request->process();
    }

    /**
     * Returns the original request parameters in the current query string as an array mapping
     * query parameter names with values. The result of this function will not be affected
     * by any modifications to `$_GET` and will not include parameters in `$_POST`.
     *
     * @return array
     */
    public static function getRequestParametersGET()
    {
        if (empty($_SERVER['QUERY_STRING'])) {
            return array();
        }
        $GET = UrlHelper::getArrayFromQueryString($_SERVER['QUERY_STRING']);
        return $GET;
    }

    /**
     * Returns the URL for the current requested report w/o any filter parameters.
     *
     * @param string $module The API module.
     * @param string $action The API action.
     * @param array $queryParams Query parameter overrides.
     * @return string
     */
    public static function getBaseReportUrl($module, $action, $queryParams = array())
    {
        $params = array_merge($queryParams, array('module' => $module, 'action' => $action));
        return Request::getCurrentUrlWithoutGenericFilters($params);
    }

    /**
     * Returns the current URL without generic filter query parameters.
     *
     * @param array $params Query parameter values to override in the new URL.
     * @return string
     */
    public static function getCurrentUrlWithoutGenericFilters($params)
    {
        // unset all filter query params so the related report will show up in its default state,
        // unless the filter param was in $queryParams
        $genericFiltersInfo = DataTableGenericFilter::getGenericFiltersInformation();
        foreach ($genericFiltersInfo as $filter) {
            foreach ($filter[1] as $queryParamName => $queryParamInfo) {
                if (!isset($params[$queryParamName])) {
                    $params[$queryParamName] = null;
                }
            }
        }

        return Url::getCurrentQueryStringWithParametersModified($params);
    }

    /**
     * Returns whether the DataTable result will have to be expanded for the
     * current request before rendering.
     *
     * @return bool
     * @ignore
     */
    public static function shouldLoadExpanded()
    {
        // if filter_column_recursive & filter_pattern_recursive are supplied, and flat isn't supplied
        // we have to load all the child subtables.
        return Common::getRequestVar('filter_column_recursive', false) !== false
            && Common::getRequestVar('filter_pattern_recursive', false) !== false
            && !self::shouldLoadFlatten();
    }

    /**
     * @return bool
     */
    public static function shouldLoadFlatten()
    {
        return Common::getRequestVar('flat', false) == 1;
    }

    /**
     * Returns the segment query parameter from the original request, without modifications.
     *
     * @return array|bool
     */
    public static function getRawSegmentFromRequest()
    {
        // we need the URL encoded segment parameter, we fetch it from _SERVER['QUERY_STRING'] instead of default URL decoded _GET
        $segmentRaw = false;
        $segment = Common::getRequestVar('segment', '', 'string');
        if (!empty($segment)) {
            $request = Request::getRequestParametersGET();
            if (!empty($request['segment'])) {
                $segmentRaw = $request['segment'];
            }
        }
        return $segmentRaw;
    }

    private function renameModuleAndActionInRequest()
    {
        if (empty($this->request['apiModule'])) {
            return;
        }
        if (empty($this->request['apiAction'])) {
            $this->request['apiAction'] = null;
        }
        list($this->request['apiModule'], $this->request['apiAction']) = $this->getRenamedModuleAndAction($this->request['apiModule'], $this->request['apiAction']);
    }

    /**
     * @return array
     */
    private static function getDefaultRequest()
    {
        return $_GET + $_POST;
    }

    private function shouldDisablePostProcessing()
    {
        $shouldDisable = false;

        /**
         * After an API method returns a value, the value is post processed (eg, rows are sorted
         * based on the `filter_sort_column` query parameter, rows are truncated based on the
         * `filter_limit`/`filter_offset` parameters, amongst other things).
         *
         * If you're creating a plugin that needs to disable post processing entirely for
         * certain requests, use this event.
         *
         * @param bool &$shouldDisable Set this to true to disable datatable post processing for a request.
         * @param array $request The request parameters.
         */
        Piwik::postEvent('Request.shouldDisablePostProcessing', [&$shouldDisable, $this->request]);

        return $shouldDisable;
    }
}

 

Share this post


Link to post
Share on other sites

Why are you posting the Request class?  I'm not sure where this topic is headed at this point.

Share this post


Link to post
Share on other sites
Quote

I could find nothing in the file that resembles an HTTP request.

Sometimes we appear to talk past one another.

Roddy

Share this post


Link to post
Share on other sites

I read what you wrote, but I don't understand the relevance.  The task now is to identify how exactly their classes send requests to the API?  In this class, that happens in the process method.  Obviously the process method uses several other classes to do that.  This is common in OOP, you have general-purpose classes like the Proxy class, or the RequestBuilder class, which each do one thing, and things get built on top of them to do more specific things.

I'm just not sure how this relates to the rest of this topic.

Share this post


Link to post
Share on other sites

The goal is to write a PHP class that will allow me to by-pass having to use HTTP requests to access the Matomo reporting API.  As I mentioned very early Matomo is a thicket of interlacing classes and namespaces that on the surface appears impenetrable.  When i look at the following public function I am inspired because its arguments mirror one-for-one, although structured differently, the content of a Matomo HTTP request query string.

The processRequest( ) Function

public static function processRequest($method, $paramOverride = array(), $defaultRequest = null)
{
    $params = array();
    $params['format'] = 'original';
    $params['serialize'] = '0';
    $params['module'] = 'API';
    $params['method'] = $method;
    $params = $paramOverride + $params;

    // process request
    $request = new Request($params, $defaultRequest);
    return $request->process();
}

The process( ) Function

public function process()
{
    // read the format requested for the output data
    $outputFormat = strtolower(Common::getRequestVar('format', 'xml', 'string', $this->request));

    $disablePostProcessing = $this->shouldDisablePostProcessing();

    // create the response
    $response = new ResponseBuilder($outputFormat, $this->request);
    if ($disablePostProcessing) {
        $response->disableDataTablePostProcessor();
    }

    $corsHandler = new CORSHandler();
    $corsHandler->handle();

    $tokenAuth = Common::getRequestVar('token_auth', '', 'string', $this->request);
    $shouldReloadAuth = false;

    try {

        // IP check is needed here as we cannot listen to API.Request.authenticate as it would then not return proper API format response.
        // We can also not do it by listening to API.Request.dispatch as by then the user is already authenticated and we want to make sure
        // to not expose any information in case the IP is not whitelisted.
        $whitelist = new LoginWhitelist();
        if ($whitelist->shouldCheckWhitelist() && $whitelist->shouldWhitelistApplyToAPI()) {
            $ip = IP::getIpFromHeader();
            $whitelist->checkIsWhitelisted($ip);
        }

        // read parameters
        $moduleMethod = Common::getRequestVar('method', null, 'string', $this->request);

        list($module, $method) = $this->extractModuleAndMethod($moduleMethod);
        list($module, $method) = self::getRenamedModuleAndAction($module, $method);

        PluginManager::getInstance()->checkIsPluginActivated($module);

        $apiClassName = self::getClassNameAPI($module);

        if ($shouldReloadAuth = self::shouldReloadAuthUsingTokenAuth($this->request)) {
            $access = Access::getInstance();
            $tokenAuthToRestore = $access->getTokenAuth();
            $hadSuperUserAccess = $access->hasSuperUserAccess();
            self::forceReloadAuthUsingTokenAuth($tokenAuth);
        }

        // call the method
        $returnedValue = Proxy::getInstance()->call($apiClassName, $method, $this->request);

        $toReturn = $response->getResponse($returnedValue, $module, $method);
    } catch (Exception $e) {
        Log::debug($e);

        $toReturn = $response->getResponseException($e);
    }

    if ($shouldReloadAuth) {
        $this->restoreAuthUsingTokenAuth($tokenAuthToRestore, $hadSuperUserAccess);
    }

    return $toReturn;
}

The getRequestVar( ) Function

$tokenAuth = Common::getRequestVar('token_auth', '', 'string', $this->request);

I suppose, for example, that i could satisfy the values of this function manually.

Looking Forward
It would appear that I have two avenues ahead:  one, figure out how to overcome the missing format of the imported widgets; or two how to access the Matomo reporting API through direct PHP calls.  This latter is likely the more difficult route, but it is also likely to result in overall better efficiency, as it would circumvent the need for HTTP requests.

Roddy

 

Share this post


Link to post
Share on other sites

As I mentioned very early Matomo is a thicket of interlacing classes and namespaces

This is common.

on the surface appears impenetrable

I'm sure that it's pretty intimidating for a novice, I can understand that.  It's fairly well-designed and well-organized though, this is what decent OOP software generally looks like, although when you abstract everything into more and more classes it can tend to get resource-heavy.  You can notice that their documentation has a pretty large section for plugin developers to use to understand how everything works together. 

But...

The goal is to write a PHP class that will allow me to by-pass having to use HTTP requests to access the Matomo reporting API.

This is very ambitious for a novice developer.  You can write your own plugins, but you need to understand how it works and how everything fits together, and that's probably not a reasonable thing to expect from you at this point.

It would appear that I have two avenues ahead:  one, figure out how to overcome the missing format of the imported widgets; or two how to access the Matomo reporting API through direct PHP calls.

Or, three, use the existing reporting API like everyone else does.

This latter is likely the more difficult route, but it is also likely to result in overall better efficiency, as it would circumvent the need for HTTP requests.

It would, but it strikes me as odd that you're trying to micro-manage efficiency in a place like this, while it seems like you also think it's fine to include the overhead of jQuery on every page, even for the most minor things.  The two are at odds when you say you want to maximize efficiency.  If I were you, I would think long and hard about how much time and effort you're willing to put into this specific thing, which does not add anything to the overall usability of your actual site.  Your visitors won't see any difference.  You've also mentioned another piece of software that you complained about and they took offline.  If that was something similar to what you're doing, then I expect the reason it didn't work is because no one updated and maintained that software while developing the core product, and it fell behind.  You can expect something similar to happen if you try to do that, they're going to release versions and updates which will probably break your code at some point.  This is something else you need to consider when asking how much time and effort you want to spend doing something like this.

Share this post


Link to post
Share on other sites
Quote

Or, three, use the existing reporting API like everyone else does.

This would mean curtailing what I have already set out to do.  See Your Data and You and Sponsors Overview at Grammar Captive overview.  The authorization token is invisible on these pages.  What everyone else does is stupidly expose their token thus giving  to knowledgeable Matomo users full access to the reporting API.  There are things in the API that even I whom am now somewhat knowledgeable to not know how to interpret.

By the way, one must strike a balance between coding, learning to code, and user experience.  View the aforementioned sections and try to convince me that the delay is created by jQuery.  Your best logic and experimentation will not succeed.  And, mind you, I am well -open to both.

Roddy

Share this post


Link to post
Share on other sites

I don't see much on that page on your site because I'm blocking trackers, so it never downloads the piwik.js file from nudge.online.  But for some reason your site had my browser send over 1,000 requests when I loaded your home page, including multiple requests for the same image and CSS files.  That may be something to look into.

What everyone else does is stupidly expose their token thus giving  to knowledgeable Matomo users full access to the reporting API.

So, the reporting API exposes all of its data to anyone with the token, and there's no way to get that data without exposing the token?  But, you're trying to show people what data gets collected, right?  So why not just let them see everything, why does it matter if anyone knows the token or not?  If your purpose is to show people what data is collected, why are you also trying to hide the token from them?

View the aforementioned sections and try to convince me that the delay is created by jQuery.

The only delay I see is the absurd number of requests (1069) which causes the load event to fire after 22 seconds.  Keep in mind that since I'm blocking the tracking script, there are probably requests other people need that I don't also.  While it's objectively true that jQuery code runs slower than vanilla Javascript, that's not going to be perceptible to users unless you're doing tens or hundreds of thousands of operations, and even then only slightly.  I just tend to enjoy optimizing things for the sake of efficiency itself, which frequently precludes using jQuery unless I'm doing fairly complex things with it.  I don't need jQuery to find or append elements.

I just think it's kind of silly to spend a lot of time trying to squeeze performance out of one specific area, and then pay no attention to efficiency in other areas.  It looks like the vast majority of requests, for example, are for that slideshow-within-a-slideshow that you have popping up, which I close.  It's sending like 1,000 requests just for that thing, so what happens if I just close it?  All of it is wasted.  Why not send those requests when it's loading each slide instead of everything at once even though I might not even look at it?  Overall I'm getting 37 requests for Javascript files, including 2 that were blocked, 76 requests for CSS files, and 900 requests to download images.  Since many of those are duplicate requests the browser will use caching, but if I disable caching it's downloading over 6MB just for that first page.

As for the parts of that page which are loading data, I see a request to get my IP address, but it's showing an IP that is definitely not mine, and for some reason it took 1.2 seconds to get the IP address.  This should take less than 1.2 seconds:

<?php echo $_SERVER['REMOTE_ADDR'];

The IP that it's showing as mine is either from California or Utah depending on whom you believe.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×