Red Cloak Endpoint Agent Troubleshooting

integrations endpoints red cloak secureworks edr

This document provides guidance on initial agent troubleshooting steps you can take prior to reaching out to Secureworks support for assistance for issues with agent performance, connectivity to Secureworks® Taegis™ XDR, and installation. Guidance is provided for both the Windows and Linux Agent.

Windows Agent Troubleshooting ⫘

Troubleshooting Performance Issues ⫘

In order to troubleshoot performance issues like CPU, memory spike, blue screen of death (BSoD), and application crashing, provide Secureworks support the following information and logs. If the log files are too large, ask Secureworks for a file share link to upload the logs.

Provide the following Information ⫘

• The hostname of the machine
• The version the agent is running
• Amount of memory on box
• Number of CPU cores
• List of services that are currently started: net start
• A brief description of the host and its role and function
• A description of how the host is being affected—e.g., poor performance, specific application issues, etc.
• Task Manager output of the Details tab to determine which module exe is actually running and causing the issue, as they all display as Secureworks Red Cloak in the Processes tab of Task Manager. You can sort by Description to obtain this information.

Task Manager Details Tab

• Is it a VM or physical hardware?
• Verify the following Red Cloak directories are excluded in the Antivirus/Endpoint software that may be running on the host:
  • C:\Program Files (x86)\Dell SecureWorks\Red Cloak\
  • C:\Program Files (x86)\Dell SecureWorks\Ignition\

Provide the following Logs ⫘

• Windows logs, saved as event files (*.evt or *.evtx):
  • Get all types: Application, Security, System, etc.
• Red Cloak logs:
  • C:\Program Files (x86)\Dell SecureWorks\Red Cloak\*.log
  • C:\Program Files (x86)\Dell SecureWorks\Ignition\*.log
  • C:\Program Files (x86)\Dell SecureWorks\Red Cloak\*.dmp
• Crash dumps, if the endpoint has experienced a blue screen of death (BSoD):
  • If possible, configure the endpoint to create a complete memory dump, which will give us the most amount of information.
  • Provide the complete memory dump in C:\Windows\Memory.dmp.
  • Provide any recent minidumps in C:\Windows\Minidump\.
• Process Monitor logs, if requested by Secureworks support:

1. Download Process Monitor: https://docs.microsoft.com/en-us/sysinternals/downloads/procmon.
2. Launch Procmon.exe.
3. Start collection in Procmon (CTRL+E).
4. Reproduce the issue:
   • Document exact steps taken to reproduce to provide to Secureworks.
   • Note local machine time when testing started, and time zone of machine.
   • Note local time when issue occurred.
5. Stop collection in Procmon (CTRL+E) and save the log by selecting File > Save.
6. Select Events to save: All events and Format: PML.
7. Change the file name to Active_<devicename>.
8. Select OK.
9. Clear current logs in Procmon (CTRL+X).
10. Run the following command from an administrator command prompt to place the sensor into safe mode: C:\Program Files (x86)\Dell SecureWorks\Red Cloak\redcloak.exe --enter-safe-mode
11. Repeat steps 3-9 to collect and save the new log as SafeMode_<devicename>.PML.
12. Run the following command to remove the sensor from safe mode:
    C:\Program Files (x86)\Dell SecureWorks\Red Cloak\redcloak.exe --exit-safe-mode

Troubleshooting Connectivity Issues to XDR ⫘

In order to troubleshoot agent connectivity issues to XDR, provide the following details.

• Change to Red Cloak installed directory, run the following command as administrator from command prompt, and provide the output: redcloak.exe --check
• Provide the following Red Cloak logs:
  • C:\Program Files (x86)\Dell SecureWorks\Red Cloak\*.log
  • C:\Program Files (x86)\Dell SecureWorks\Ignition\*.log

Troubleshooting Installation Issues ⫘

In order to troubleshoot agent installation issues, provide the following details.

• Provide the following logs:
  • C:\Program Files (x86)\Dell SecureWorks\Red Cloak\*.log
  • C:\Program Files (x86)\Dell SecureWorks\Ignition\*.log
• Windows logs, saved as event files (*.evt or *.evtx):
  • Get all types: Application, Security, System, etc.
• How is the Red Cloak agent installed? Is it via a software deployment tool or manual install?
• List of services that are currently started: net start.
• Agent version that's being installed.
• Is the host a physical server or VDI host?

Collecting Red Cloak Debug Logs ⫘

While troubleshooting connectivity and performance issues, Secureworks support may request debug logs from the endpoint. Follow these steps to collect the Red Cloak logs in debug mode upon request from Secureworks.

1. Stop Red Cloak service.
2. Open registry editor.
3. Find the following entry:
   • Key Name: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\redcloak
   • Class Name: ImagePath
4. Edit ImagePath original value by adding --debug=2 parameter. Note two dash characters (--) before debug.

Original: C:\Program Files (x86)\Dell SecureWorks\Red Cloak\redcloak.exe" --run-service --override-root "C:\Program Files (x86)\Dell SecureWorks\Red Cloak\\

Updated with --debug=2 parameter: C:\Program Files (x86)\Dell SecureWorks\Red Cloak\redcloak.exe" --run-service --debug=2 --override-root "C:\Program Files (x86)\Dell SecureWorks\Red Cloak\\

5. Start Red Cloak service.
6. Agent starts in debug mode and writes verbose information into the log files. Nothing changes in its behavior except more information in log files, and faster file growth is expected because of this.
7. Once the log collection is completed, revert the changes. Debug mode is not recommended to be run for a longer duration than recommended by Secureworks support.

Linux Agent Troubleshooting ⫘

Troubleshooting Performance Issues ⫘

In order to troubleshoot performance issues like CPU, memory spike, and application crashing, provide Secureworks support the following information and logs. If the log files are too large, ask Secureworks for a file share link to upload the logs.

Provide the following Information ⫘

• The hostname of the machine
• The version the agent is running
• Results of the command top with Irix mode off (run top command and press Shift + i)
• Which module is consuming the most CPU: Procwall or Lacuna (around 20-30% is average)
• Results of the command cat /proc/cpuinfo
• Results of the command free -m
• What applications is the agent running?
• Is it a VM or physical hardware?
• What is the role and function of the server?
• OS and kernel information of the endpoint?
• Results of the command service --status-all | more

rc_collect Script for Performance Capture ⫘

On Linux Agent version 1.2.13.0 and above, we have installed an rc_collect script for performance captures. This can be used to capture over a duration and is used to do a snapshot of details on the agent that Secureworks support and development teams use for performance analysis. The script can be run for five minutes.

Usage: <agent_install_path>/bin/rc_collect <runtime in seconds> <output path>

Example: sudo /opt/secureworks/redcloak/bin/rc_collect

Example with time and output arguments: sudo /opt/secureworks/redcloak/bin/rc_collect 360 /var/tmp

Provide the following Logs ⫘

• For Linux Agents below version 1.2.12.0: /var/opt/secureworks/redcloak/log
• For Linux Agents 1.2.12.0 and above: /opt/secureworks/redcloak/log

Troubleshooting Connectivity Issues to XDR ⫘

For Agent Connectivity issues to XDR, provide the following details.

• Connectivity Check Output:
  • For Linux Agents below version 1.2.12.0: /var/opt/secureworks/redcloak/bin/redcloak --check
  • For Linux Agents 1.2.12.0 and above: /opt/secureworks/redcloak/bin/redcloak --check
• Linux Red Cloak Logs:
  • For Linux Agents below version 1.2.12.0: /var/opt/secureworks/redcloak/log
  • For Linux Agents 1.2.12.0 and above: /opt/secureworks/redcloak/log

Troubleshooting Installation Issues ⫘

In order to troubleshoot agent installation issues, provide the following details.

• OS version of the endpoint
• Agent version that's being installed
• Provide complete install logs with any errors:
  • CentOS/Red Hat/Oracle: yum localinstall <redcloak_filename>.rpm (complete log with any errors)
  • Ubuntu: sudo apt install PATH_TO_DEB (complete log with any errors)
• Provide the following logs:
  • For Linux Agents below version 1.2.12.0: /var/opt/secureworks/redcloak/log
  • For Linux Agents 1.2.12.0 and above: /opt/secureworks/redcloak/log
• How is the Red Cloak agent installed? Is it via software deployment tool or manual install?

Collecting Red Cloak Debug Logs ⫘

While troubleshooting connectivity and performance issues, Secureworks support may request debug logs from the endpoint. Follow these steps to collect the Red Cloak logs in debug mode upon request from Secureworks.

1. From the command line, execute:

vi /opt/secureworks/redcloak/bin/redcloak_start.sh

2. Within vim (or vi), find the following line and add --debug=2:

Original: ${prefix}/bin/redcloak --run-service --override-root "${prefix}" > /dev/null 2>&1 &

Updated with --debug=2 parameter: ${prefix}/bin/redcloak --run-service --debug=2 --override-root "${prefix}" > /dev/null 2>&1 &

3. Save and quit by hitting ESC and typing :wq!
4. Restart Red Cloak service: systemctl restart redcloak
5. Agent starts in debug mode and writes verbose information into the log files. Nothing changes in its behavior except more information in log files, and faster file growth is expected because of this.
6. Once the log collection is completed, revert the changes. Debug mode is not recommended to be run for a longer duration than recommended by Secureworks support.