ArcGIS Server - missing Output Directory

This morning I found that services on one of our ArcGIS 10.1 servers were extremely slow, causing timeouts for users. I found that javaw.exe was consuming 100% CPU across all cores. This does not happen very often, in fact it must be a year since last time something like that happened. Anyway, I tried restarting the ArcGIS Server service, but CPU-consumption immediately went back to 100%. Next I tried to restart map services to see if there was one particular that caused the problem. Unfortunately I couldn’t restart the services once they were stopped. Time to read logs.

Two errors would constantly be logged to the ArcGIS Server-log:

<Msg time="2014-12-12T15:59:18,46" type="SEVERE" code="9503" source="Soap" process="10604" thread="24" methodName="" machine="ARCGIS2" user="" elapsed="">Service 'mymapservice.MapServer' not found. Check if the service exists and is started.</Msg>

<Msg time="2014-12-12T15:59:18,60" type="SEVERE" code="9003" source="Rest" process="10604" thread="18" methodName="" machine="ARCGIS2" user="" elapsed="">Unable to process request. Error handling service request : Could not find a service with the name 'MapServer/mymapservice2' in the configured clusters. Service may be stopped or ArcGIS Server may not be running.</Msg>

There was nothing useful in the ArcGIS/services-logs. Not good.

Next I tried publishing the services again (overwrite existing) – it got as far as to “upload service definitions” before failing. Tried deleting the map service and publishing again – it still wouldn't work. Then I had a look at the PublishingTools service it self. It looked all right, but when I restarted it I got an error message saying it could not find c:\arcgisserver\directories\arcgisoutput2 folder. This didn't make sense as ArcGIS Manager showed the following valid configuration:

Time to go behind the GUI and look at config files. The publishing tools service config is located at: C:\arcgisserver\config-store\services\System\PublishingTools.GPServer\PublishingTools.GPServer.json

.. and there was my my problem:

"outputDir": "C:\\arcgisserver\\directories\\arcgisoutput2".

I’m not too keen on altering ArcGIS Server config files by hand so I tried going back to the service definition and just selected the same directory:

..and then “save & restart”. That did the trick – and the config-file was updated. Basically it turns out all my services were configured to use c:\arcgisserver\directories\arcgisoutput2 despite of what ArcGIS Server Manager displayed, so I had to repeat this for every service. I suppose the reason was that I moved the ArcGIS-directories around a while ago as part of a cleanup job, and left the arcgisoutput2-folder behind for a while before deleting it thinking it was no longer in use.