Jump Clients offline

We user both RS & PRA. 

I had this issue for many years and it was a case of EURICA!!!  in the middle of the night that I figured it out.  Support could not figure the issue after several upgrades they were still baffelled as to why I only had the issue.  I knew it couldn’t be just me.  During the upgrade it appears that the Jump Clinet/Jump Points that the server was on had someone connected to the conse with an active session.  So to test this theory on 24.1.1 upgrade I went into the systems of 193 servers and logged off anyone that was logged onto the console. 

Low and beholde the upgrade ran 100% successful for the first time ever Jump Points and Jump Clinents.  I completed 3 upgrades since then and used the same process and all upgrades are 100% successful on windows.  Mac and Linux had recent upgrade issue that had to be resolved manually.  

Now that we have the Endpoint Automation process I can send out the command to log out the console minutes prior to the upgrade.  See if this could potentially be the issue.  

I work primarily with Remote Support - not PRA, but I’d imagine the Jump Clients are similar.  It’s strange, I’ve never seen an update process even have the step for deleting and then upgrading.  Usually it just “upgrades”.  By any chance, were there different jump installers used for different computers?  If so, did you check the box that talks about “Elevated Installs” in the jump client creation window? (See Below)

Logically speaking, if you’re in a situation where you have jump clients that cannot be updated for whatever reason (i.e. cant), it would make sense for the jump clients to uninstall themselves as they will no longer function.  In my experience (and someone correct me if I’m wrong), but the Appliance Software Version must match the Jump Software Version, or it wont work.  However, once it uninstalls itself, it should show up as “uninstalled” as an entry in the rep console. (See Below):

So while it can be frustrating that there is an entry for it, it is there to show you that it has been uninstalled - which you may not have wanted to happen.  So when you install a new jump client, there will indeed be a duplicate.  If the Appliance wasn't “notified” of an uninstall (which I’ve seen happen on Macs), then yes, there will be duplicates.  Thankfully, you can look at the “last seen” date, and the older one will almost always be the old one, so you can generally safely delete it.

The most upvoted idea in the Ideas Portal is for the Jump Client management tool/removal tool:

https://beyondtrust-public.ideas.aha.io/ideas/A01-I-1713

Reboot Issue:  There is no area in the /appliance interface to reboot?  In the /appliance of Remote Support, on the Status page, towards the bottom, there is a button to reboot the appliance.  Though, I’ll say this… in the decade we’ve had BTRS/Bomgar, we’ve never had to reboot any of our appliances (we have more than one) - not even once.

Support Issue:  I highly recommend you use the Chat function.  I use it pretty frequently when I need help with things.  The support is top notch.  I’ve personally never had issues with their support.  They’ve literally hand-held me through some real basic stuff.

Thanks @pcoats 

I will have to give that a try. Although I would assume at my organization all 100 servers have someone logged in. 
------------------------------------------------------------------ 

@LayerZeroIssue 
I only know it was uninstalled because its no on the app list any longer. Also, our IT inventory picked up the uninstall action and mapped it to the same time frame I upgraded the appliance. 

In the console, they all show as Active cOffline] 
 

When I reinstall, those entries are still there, and there is a new entry that shows online with the same name, IP, everything. 


And yea, it’s crazy there is not a restart option on the appliance. I am 98% sure there used to be one. 
Maybe it’s gone in the new version? I also saw in screenshots there is supposed to be a “Health” button next to “basics”. I verified this account was an admin. Not sure what else to do, and support still hasn’t contacted me.

I thought they shut down the chat feature. I can’t find it anymore. 

 

Huh, I’ve had this issue for the past 4 upgrades I’ve done, and I haven’t been able to figure out what the cause of it was. I’ll have to do some more experimenting, as we use RS for a PAM-esque solution and losing the installed jump clients is a pain for those teams.

I schedule my updates with no users logged in to try and ensure that within the given time frame, all the end points are updated by the time I send out notification to re-login.

It’s for reasons such as this - I’ll also terminate users sessions manually prior to updating once my “scheduled window” begins.

Hello,

I can relate to your issue with a strange behaviour after upgrading the appliances: I need to restart the Beyond service on the jumpoints otherwise the endpoints appear offline.

@Mark Gunnett Nice to know it is not just me. I have no idea what it could be. I looked in event viewer, even reviewed SIEM for msiexec commands for an uninstaller, couldn’t find anything

@K_jf  Logged into an endpoint or logged into an endpoint with beyondtrust? I was the only one using BeyondTrust at the time. 

@RazVan I cannot restart the service as the program is uninstalled on the endpoint. All services removed, However, I do see a registry entry in HkeyLM - Software - Bomgar - System_id - %Value%

@Jsizzle45 

This issue can occur in some environments that are more restricted than others.  The process of the jump client upgrading are below.

Using the online devices logged in user context (system user) in most cases, the appliance goes through these general steps (there are more nuances than this and there is a KB that goes over the client process):

1. Stops the service
2. sends an uninstall package
3. sends an install package for the new version
4. Starts the service

If any of these processes are interrupted, it will proceed to the next step in the process.  This is just one of the ways that can cause offline clients and duplicated clients.  If your environment does not allow uninstalls - then this process will get interrupted and proceed to install the new version.  In most instances regarding this issue, a ticket would be recommended.  Another solution we offer is the ability to have full control of your upgrades regarding jump clients and Access/rep consoles.  You can log a ticket with support and they can provide you the installers for the version you are moving to.  Once this is flagged for your software, it will always be available, and you shouldn’t need another ticket to grab the software.  

Once your software has been built to include them - To access them, is via manual download. Go to your appliance -> update tab -> click appliance download key at the bottom -> copy everything in that box -> go to update.bomgar.com -> paste the key and click download -> you should see an additional option for files/supplemental installers. Click that to download the installers for your rep console and jump clients.

When you are ready to deploy the clients, grab the MSI command from /login -> my account to install the MSI's. If you want it to auto-update - include the SHOULDAUTOUPDATE=1 string at the end of the MSI command.

For jump clients, you will go to /login -> jump -> set all the parameters you want -> hit create -> Select MSI and it will reveal the string. DO NOT download or use the MSI from /login -> jump. We only need the string it produces. Then use that string with the updated jump client for your next version and deploy them out after removing all previous jump client versions via MSI.

You can remove jump clients via the MSI by changing /i to /x and running it twice. Once to remove all jump clients via mass deployment or single machine (works both ways). The second /x is to verify it has purged all registry remnants. Then mass deploy with /i.

Something also to consider, if the system uptime on the devices is high (over a year) this can prevent some services from stopping/starting.  

Another instance that can cause duplicates is if you have a deployment tool and not using a flag to show a client is installed.  

Below is the jump client guide as well as all the mass deployment options, we have guides for

Jump client guide
https://www.beyondtrust.com/docs/remote-support/documents/features/rs-jump-clients.pdf

Mass deploy jump clients and avoid duplicates 
https://beyondtrustcorp.service-now.com/csm?id=kb_article_view&sysparm_article=KB0017080

Spoke with support. 
They stated that this can sometimes be caused by various EDR solutions (We are using S1)

There are no logs or events to look at after the fact to verify this.
Next time there is an upgrade available, I am to download an offline version of the software from the appliance. (An option near the upgrade button on the appliance when applicable) 

I will then install the client on a VM or some other machine. I will be able to get the hash of the installer and add it to the whitelist in my EDR. 

If I need help I can reach out to my account rep to ask about white glove service. 

There is no way to match an installer with the machines that are already in appliance. This is a the way its supposed to work so that users can have multiple jump groups and installers. 

Support also gave me this powershell query. 

Get-ItemProperty HKLM:\Software\Microsoft\Windows\CurrentVersion\Uninstall\* | Where-object {$_.DisplayName -ne $null -and $_.SystemComponent -ne "1"} | select DisplayName, Publisher, DisplayVersion, UninstallString | Where {$_.DisplayName -match "Jump Client"}
 

@RKarnitz , Thanks for the information about how the jump client upgrades work. Unfortunately, I’m not sure I can use the installer pre-downloads to upgrade all of the jump clients, as not all of the endpoints are managed by any kind of inventory system in my environment.

Since you indicate that upgrades are super sensitive, maybe I’ll try to utilize the new upgrade pause feature for jump clients to only upgrade after I’m done restarting the appliance during our maintenance window.