ArcGIS Server - missing Output Directory

This morning I found that services on one of our ArcGIS 10.1 servers were extremely slow, causing timeouts for users. I found that javaw.exe was consuming 100% CPU across all cores. This does not happen very often, in fact it must be a year since last time something like that happened. Anyway, I tried restarting the ArcGIS Server service, but CPU-consumption immediately went back to 100%. Next I tried to restart map services to see if there was one particular that caused the problem. Unfortunately I couldn’t restart the services once they were stopped. Time to read logs.

Two errors would constantly be logged to the ArcGIS Server-log:

<Msg time="2014-12-12T15:59:18,46" type="SEVERE" code="9503" source="Soap" process="10604" thread="24" methodName="" machine="ARCGIS2" user="" elapsed="">Service 'mymapservice.MapServer' not found. Check if the service exists and is started.</Msg>

<Msg time="2014-12-12T15:59:18,60" type="SEVERE" code="9003" source="Rest" process="10604" thread="18" methodName="" machine="ARCGIS2" user="" elapsed="">Unable to process request. Error handling service request : Could not find a service with the name 'MapServer/mymapservice2' in the configured clusters. Service may be stopped or ArcGIS Server may not be running.</Msg>

There was nothing useful in the ArcGIS/services-logs. Not good.

Next I tried publishing the services again (overwrite existing) – it got as far as to “upload service definitions” before failing. Tried deleting the map service and publishing again – it still wouldn't work. Then I had a look at the PublishingTools service it self. It looked all right, but when I restarted it I got an error message saying it could not find c:\arcgisserver\directories\arcgisoutput2 folder. This didn't make sense as ArcGIS Manager showed the following valid configuration:

Time to go behind the GUI and look at config files. The publishing tools service config is located at: C:\arcgisserver\config-store\services\System\PublishingTools.GPServer\PublishingTools.GPServer.json

.. and there was my my problem:

"outputDir": "C:\\arcgisserver\\directories\\arcgisoutput2".

I’m not too keen on altering ArcGIS Server config files by hand so I tried going back to the service definition and just selected the same directory:

..and then “save & restart”. That did the trick – and the config-file was updated. Basically it turns out all my services were configured to use c:\arcgisserver\directories\arcgisoutput2 despite of what ArcGIS Server Manager displayed, so I had to repeat this for every service. I suppose the reason was that I moved the ArcGIS-directories around a while ago as part of a cleanup job, and left the arcgisoutput2-folder behind for a while before deleting it thinking it was no longer in use.

Tuning GeoServer MSSQL data stores

At work we recently installed some new layers on our GeoServer 2.4.4 which queried for spatial data from a MSSQL 2008r2 database. Unfortunately we soon realized we had some serious performance issues. I tried the usual tricks - updating statistics, checking indexes etc. Microsofts Database tuning engine advisor eventually showed that things were decent - yet we still expericnced poor performance.

I then tried modifying settings for the data store. For MSSQL the default settings are:

max connections:	10
min connections:	1
fetch size:	1000
connection timeout:	20
validate connections:	enabled
Use native geometry serialization:	disabled

I ended up changing the settings to

max connections:	20
min connections:	5
fetch size:	5000
connection timeout:	20
validate connections:	disabled
Use native geometry serialization:	enabled

Performance got a real nice boost. I estmate that things are at least twice as quick now, and This is especially noticeable when the MSSQL server is also quite busy with other tasks (especially heavy disk/network activity during backup).

Especially the "validate connections" parameter seems to have a noticeable impact. According to the documentation this parameter increases the risk of client errors. I suppose I will have to monitor the services for a while to see if they remain stable.