webread - not behaving per docs ?

According to the mathworks documentation,

On Windows®, if no proxy is specified in MATLAB preferences, webread, webwrite, and websave use the proxy set in the Windows system preferences.

Now, lets say I'm trying to read a url here, which is internal company information.

http://10.65.15.180/blah/blah/blah

The windows proxy settings at my company have a bunch of exclusions. One of the exclusions is;

10.65.*.*;

Matlab gives me back the following error.

The server returned the message: "Proxy authentication required". What I want to know is, why? Is this is expected behavior?

I submit that matlab is ignoring the list of proxy exclusions, because it should never reach the proxy server if it was respecting the settings. Anyone able to shed light on the behaviour here?

[Simon Parten]

I think I have proof that matlab does not respect the proxy server exclusions; if I set the proxy settings in matlab to point directly at my internal website - i.e. in matlab preferences, use this; http://10.65.15.180 then I can use webread as expected.

I guess I'd like to submit an enhancement request to the mathworks, that webread should respect the list of exclusions in the default windows proxy settings. Anyone know how I can do that ?

[Rakesh Chavan]

Hi,

This looks like a bug in the documentation for MATLAB R2016a wherein it does not list a limitation with regards to MATLAB not honoring system proxy exceptions.

To work around this limitation you could consider adding your own logic to +matlab/+internal/+webservices/HTTPConnector.m to determine which proxy should be used for which host.

Meanwhile you could also write to MathWorks Technical Support in case you need further assistance. The following link can be used: http://www.mathworks.com/support/servicerequests/create.html

-

Rakesh

Answer 1

[Kurt Fischle]

I had the exact same problem; I applied Rakesh's workaround suggestion by changing line 156 in C:\Program Files\MATLAB\R2016a\toolbox\matlab\external\interfaces\webservices\restful\+matlab\+internal\+webservices\HTTPConnector.m to

if isempty(connector.Proxy.Host) || true

This disables the proxy for all webread requests. Now I can use webread to read content from the internal server.

If you need to use webread both for external and internal addresses, you would need to add some more code.

[Kurt Fischle]

...a slightly more elegant solution would be to change HTTPConnector.m as follows:

function proxy = getProxySettings(url)
% Get proxy settings from MATLAB preferences panel. If settings are not
% found in MATLAB then system proxy settings will be used.
% This calls into C++ to get the settings which 
% returns a struct with fields {"Host", "Port", "Username", "Password"}
proxy = matlab.internal.webservices.getProxyInfo(url);
%*** Modification: do not use proxy for internal URL **********************
if ~isempty( strfind( url, 'YOUR_INTERNAL_SERVER_NAME' ) )
	proxy.Host = '' ;
end
%*** End of modification **************************************************
end

Then webread() works for both internal and external addresses.

But of course the easiest and most elegant solution would be if MATLAB would just apply the Windows proxy exceptions. I'm a bit disappointed that this has still not been fixed in MATLAB 2020a; or am I the only one who is trying to use webread() in a company network where a proxy is required for external addresses, but not for internal addresses?

Answer 2

[Nikolay Golovin]

I've had similar problem - very slow execution of webwite for internal adresses.

That works for me

in C:\Program Files\MATLAB\R2019a\toolbox\matlab\external\interfaces\webservices\restful\+matlab\+internal\+webservices\HTTPConnector.m on string 697

function proxy = getProxySettings(url)
% Get proxy settings from MATLAB preferences panel.


proxy = struct( ...
    'Host', '', ...
    'Port', [], ...
    'Username', '', ...
    'Password', '');

%*** Modification: do not use proxy for internal URL **********************
if ~isempty( strfind( url, 'YOUR INTERNAL ADRESS.' ) )
	proxy.Host = '' ;
else
%*** End of modification **************************************************

    if usejava('jvm')
        % Get the proxy information using the MATLAB proxy API.
        % Ensure the Java proxy settings are set.
        com.mathworks.mlwidgets.html.HTMLPrefs.setProxySettings

        % Obtain the proxy information.
        url = java.net.URL(url);
        % This function goes to MATLAB's preference panel or (if not set and on
        % Windows) the system preferences.
        javaProxy = com.mathworks.webproxy.WebproxyFactory.findProxyForURL(url);
        if ~isempty(javaProxy) 
            address = javaProxy.address;
            if isa(address,'java.net.InetSocketAddress') && ...
                javaProxy.type == javaMethod('valueOf','java.net.Proxy$Type','HTTP')
                proxy.Host = char(address.getHostName());
                proxy.Port = address.getPort();
                    % If proxy information came from MATLAB settings, also get the
                    % username and password.  If not, we can only talk to unauthenticated
                    % proxies.
                mwt = com.mathworks.net.transport.MWTransportClientPropertiesFactory.create();
                if ~isempty(mwt.getProxyHost())
                    proxy.Username = char(mwt.getProxyUser());
                    proxy.Password = char(mwt.getProxyPassword());
                end
            end
        end
    else
        % The Java JVM is not running. The MATLAB proxy information is obtained
        % from the MATLAB preferences using Java. Return the default structure
        % containing empty values.
    end
end
end