8603 - PowerScribe 360 - Troubleshooting the SUS Server

QUESTION:

What information is available for Speech Profile Management and Speech Utility Server (SUS)?

ANSWER:

Speech Profile Management and Speech Utility Server (SUS) details are provided below:

Introduction

Profile management is an important part of making the dictators and the site successful using PS360 Reporting.

As users dictate throughout the day, for each report, a DRA file generates. DRA files are stored on the user workstation until the user saves their speech files to the server. These DRA files are necessary for Optimization jobs (ACO/LMO) to run successfully.

If a user trained a word today on workstation and moves to a different workstation before the nightly LMO runs, the trained word does not follow the user. Once the LMO runs, the trained word will be available system wide. Only DRA files that contain 30 seconds of audio and 2 voice corrections will input to the ACO process.

Keyboard edits are currently not included in corrections for adaptation (ACO). Therefore, if a provider or an editor uses the keyboard to edit mis-recognition errors, the mis-recognition errors will not be used for ACO.

Dragon Speech Profiles are stored on the webserver, in the folder location of
X:\Nuance\DragonUsers. This is also a shared folder \\<webserver>\DragonUsers.

There are 2 ways for the locally saved profile data to get to the server.

Users can save their speech data manually, by going to Speech > Save Speech Files in the Client, or when the user logs off of the Client, depending on preferences.

Users must complete the logoff for speech files to upload. If the user leaves their sessions open and lets the inactivity timer log to time out, the user speech files will not be saved.

Profile sizes can vary. Larger profiles do not necessarily cause latency. PACS systems typically control the logging out of the PowerScribe 360 Reporting Client, however the specific log out method should be verified with the PACS vendor.

When logging out of the Client, the application will perform a check of the size of the profile stored on the server. If the profile is greater than 500 MB, data will NOT be copied form the workstation to the server. During this time, the user will see a splash screen saying "saving speech files" for about 10 seconds.

If the application determines the server profile is less than 500 MB, the application will create a new session folder on the server and upload the profile data from the workstation. It will then delete the profile data from the workstation.

If it is noted that a user's profile goes through ACO (ACO analyzes a provider speech/acoustic patterns and applies this analysis to the acoustic model.) as scheduled, but the data keeps building up on the workstation, the site admin should change the frequency of ACO for that user to be more often. For example, every 2 or 3 days. This will allow ACO to run more often, therefore trimming the profile more often and allowing the user to upload data from the workstations more effectively.

This should not be done for the entire site, but only for dictators who read a great deal of reports per day.

Verify the SUS Services Are Running

SPARK core services
SPARK host manager service
SPARK speech node manager
SQL Server (SQLEXPRESS)
Speech utility server

Verifying Jobs in Spark DB

If SUS services are running, you can verify that jobs are being created in the Spark Database by running this query:

Select * from Job

If there are jobs, check the state of the jobs, and determine if there is a job that is not processing.

Multiple Jobs State 1: If there are multiple jobs in state 1, then there is a job marked as active that is not processing. Identify the job, and set the State to 4 in SQL manually. Be careful though, as multiple nodes will cause multiple jobs to have a state of 1.
Created from template: Job is scheduled but has not yet run.
There is not enough data to perform the ...Adaptation: Not enough data has accumulated for the user to begin an adaptation.
Adapter Progress: The job is being processed by Dragon NaturallySpeaking engine.
Processing...: The job is being processed by the SUS Server (File copy, etc).
Conversion attempt failed: There was an error processing the job. See the next section.
Set to Failed state by HostManager: Job failed to trigger and should be automatically re-queued.

If there are no jobs

If there are no Jobs, check the JobTemplate table:

Select * from JobTemplate

If there are no JobTemplates, there is an issue communicating with RAS.
Review SUSServiceLog.txt and reinstall SUS if necessary.
If there are JobTemplates, confirm the ACO/LMO intervals in RadPortal.
Run the update script below:

Update JobTemplate

Set LastRun = LastRun - 7

Verify that jobs begin to process

Check the RadPortal Settings

There are specific settings in RadPortal that can affect profiles and SUS operation.

RadPortal > Setup > System > Configuration > Speech Utility

This value should be the UNC path of the DragonUsers folder.

Reviewing Exception / Failure Messages

If you find a profile that is still failing ACO or LMO, after running the Speech Profile Remediation Utility, review the stack trace to determine the cause.

All exceptions are logged to RadPortal. Jobs may be completing successfully, but finishing with a state of 5 (Not Enough Data), which do NOT show in RadPortal. Exceptions are also logged to the Windows Event Viewer.

Event Logging

SUS will create a new Event Log in the Window Event Log, where all SUS related events will be logged. The name of this log will be Speech Utility Server. The RD agent can monitor this log for all SUS activity. Each entry in the event log will have an Event Id and an Event Category associated with it. Additionally, descriptive text will be logged as well. The text will typically provide a higher level description of the error. The log files associated with a job or event source may need to be consulted to provide additional details of the event.

There will be multiple event sources writing to the Speech Utility Server event log. Each source is a separately running process.

Source Name	Process Name	Sever	Description
SUS Speech	Nuance.ConversionServer.ClientApp.exe	Speech Node	The Speech Node service, which performs SUS tasks on user profiles.
SPARK Core	SPARKService.exe	SUS Server	The core service responsible for scheduling jobs to SUS Speech Nodes.
PS360 Sync	SUSService.exe	SUS Server	The SUS service which synchronizes changes on the PS 360 system with the SPARK database.
SPARK Host Mgr	HostManagerService.exe	SUS Server	The service which monitors SUS Speech Nodes that may require a service restart, and looks for long running jobs.
SPARK Node Mgr	NodeManagerService.exe	Speech Node	The Node manager which starts the SUS Speech Node, and restarts it when commanded by the SPARK Host Mgr.

The following Event Ids are defined. A given event Id can be generated from any event source. The event source/category provides context, and the event description provides details.

For example, both the Speech Node and Host Manager may post a JobFailed event. The event category adds more context as to the why, and the textual description in the event log provides the human readable reason.

Event ID	Descriptive Name	Description
0	None	An event Id is not needed. Typically used for a general informational message. The Event text provides any descriptive information.
1	JobStarted	A job was started by the Event Source
2	JobComplete	A job finished without error
3	JobFailed	A job encountered an error
4	JobCancelled	A running job was canceled by the system
5	JobPostponed	A job was postponed by the system
100	ServiceStarted	A SUS service has started by the Windows Service Control Manager
101	ServiceStopped	A SUS service was stopped by the Windows Service Control Manager
102	ServiceFailed	A service failed to start, or unexpectedly ended.

The following Event Categories are defined.

Event ID	Name	Description
0	None	An event category is not needed. The Event text provides any descriptive information.
1	ACO	The event is associated with a SUS ACO task.
2	LMO	The event is associated with a SUS LMO task.
100	ConfigurationError	A possible configuration error was detected. If this error is generated from every job, it is most likely a system wide error. If the error is associated with only a specific user or only a few jobs, the problem could be environmental, or something is wrong with the user's profile.
500	ProfileAccess	Indicates trouble accessing the profile. This issue is typically seen when the Speech Node is attempting to copy the user's profile to the temporary location, or when the user's profile is being updated after a task completes.
900	TimeOut	A timeout error occurred when performing an operation, or a job has been in the "running"? state too long without reporting progress updates.
901	NodeReset	A node was reset, which caused the event log to be generated.
999	UnhandledException	An unexpected failure has occurred.
5000	DNS SDK Error	The DNS SDK reported an error to the Speech Node.

SUS Log Files

SUS Log files are located in:

2003 - C:\Docs and Settings\All Users\Application Data\Nuance
2008 - C:\ProgramData\Nuance

Log file locations can vary. The log file location is specified during the installation. If the log files are not installed in a specified installation location, you can find the files typically in the X:\Nuance directory.

SUS log Names

Spark Core Services
1. SPARKServicesLog.txt
Spark Host Manager Service
1. HostMgrLog.txt
Spark Speech Node Manager
1. SUSClientLog.tx
2. SUSNodeMgrLog.txt
Speech Utility Server
1. SUSServiceLog.txt

Process to Review Logs

For problematic users, get the username from RadPortal
In the Jobs folder, locate the logs by matching the time of the logs with the time of the exception from RadPortal.

The log file naming convention is defined as the following:

8(30)_nwhite_1_workernode.log.zip
8 - JobID from the Job Table in the Spark DB
30 - Retry Number (jobs are repeated when there is not enough data)
nwhite - Username
1 - Job Type. Can be 1 for ACO, 2 for LMO
Workernode - Log Type. Can be Dragon or Workernode.

Review the Dragon log first.
- If the file is invalid, then Dragon likely never was initiated, and the error is in the SUS processing.
  1. You must review the log like any other Dragon log. Start from the bottom, scrolling up until you find an exception. Most of the time, the errors will point to a corrupt or missing file in the voice profile. See the Speech Profile section below for more detailed information.

If the source of the error was not found in the Dragon log, review SUSClientLog.txt
- You can search the log file for errors in Notepad by hitting Ctrl-F and searching for error. It is less likely to have errors in SUSClientLog.txt, as it would likely affect ALL users.
1. If you receive errors on all SUS jobs, this could be an issue with the SUS installation.
2. Review a SUSClientLog.txt from the Jobs folder for one of the failed jobs.
  - If the error appears to be an issue with the PSExec use the process above to change the SUS connection method to the application server.
  - If it is an authenticating error, you will see a line that access is denied.
If you continue to receive the error with the confirmed credentials, UAC could be the problem.
1. Disable UAC for the authenticating account, and reboot the web server when time is allowed.
2. If the Access Denied error is not resolved, the add/edit the following registry key and reboot the web server:
  HKLM\Software\Microsoft\Windows\CurrentVersion\Policies\System
  
  Add a DWORD value of LocalAccountTokenFilter and modify the value, setting it equal to 1.
3. Antivirus can also cause this if the proper exclusions are not put into place.

Common Errors

SUS cannot connect to Application Server

If you see an event indicating SUS cannot connect directly to Application Server or Cluster Access Point (CAP) "psexec.exe cannot connect to remote source with either a Domain or Local count. Error 2250", you should reconfigure SUS to not use PXEXEC (V1.x and V2.x).

System.IO.IOException

With the System.IO.IOException, you may encounter, the specified network name is no longer available or the parameter is incorrect, as well as many potential exceptions.

These errors are related to SMB2, which is how SUS is accessing shares. This must be disabled by adding/confirming keys in HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\Lanmanworkstatio n\Parameters:

DWORD DirectoryCacheLifetime to 0
DWORD FileNotFoundCacheLifetime to 0
DWORD These setting can also be set at a user to 0

Nuance.ConversionServer.Common.ConverterException

Nuance.ConversionServer.Common.ConverterException: Write Test of profile location failed. An UnauthorizedAccessException occurred while testing folder access.

This issue is related to improper SHARE permissions (not security permissions on either the DragonUsers or DragonUsersTemp folder.
To adjust the SHARE permissions, right-click DragonUsers(Temp) folder > Properties > Sharing > Advanced Sharing > Permissions, and set the appropriate share permissions. The user setup on SUS to access the webserver must have FULL permissions.